di-agent-knowledge-engine-streamsets

name

di-agent-knowledge-engine-streamsets

description

Reference for StreamSets Data Collector engines and StreamSets environments — StreamSets environment configuration, StreamSets engine deployment (Docker/Podman), StreamSets job execution, StreamSets engine communication methods (tunneling/direct), StreamSets high availability and failover, StreamSets monitoring and resource management. Use ONLY when the user explicitly mentions StreamSets, Data Collector, or a StreamSets-specific concern.

StreamSets Engine

Flow authoring: This skill covers engine infrastructure and operations only. For building or modifying StreamSets flows, use your path's flow authoring skill.

When to Use This Skill

• Configuring or troubleshooting StreamSets environments and engines
• Engine deployment (Docker/Podman), communication methods (tunneling vs. direct)
• Job lifecycle: execution, monitoring, failover, offset management
• High availability setup and resource management

Engine Characteristics

• Container-based: Runs as Docker/Podman containers in your network
• Data ownership: All processing occurs within your infrastructure
• Continuous processing: Jobs run continuously, processing data as it arrives
• High availability: Multiple engines support failover and load distribution
• Flexible communication: Tunneling (default) or direct connection methods

Key Concepts

• Environments: Configure engines and compute resources for projects
• Engines: Data Collector containers that execute flows
• Jobs: Flow executions that run continuously on engines
• Offsets: Track processing progress for resumable execution
• VPCs: Virtual CPUs allocated to engine containers

Architecture

• Browser ↔ watsonx.data integration ↔ Engine (in your network)
• Engines process data locally, send metrics to watsonx.data integration
• Jobs fail over automatically between engines (6.4+)
• Resource thresholds prevent engine overload

Quick Start

1. Create environment → environments.md
2. Run engine → engines.md
3. Execute job → jobs.md

References

• Environments - Environment creation and configuration
• Engines - Engine deployment, management, and monitoring
• Jobs - Job execution and lifecycle management
• Technical Reference - System requirements, commands, and troubleshooting