This skill should be used when users need to create or fix Redpanda Connect pipeline configurations. Trigger when users mention "config", "pipeline", "YAML", "create a config", "fix my config", "validate my pipeline", or describe a streaming pipeline need like "read from Kafka and write to S3".
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
This skill should be used when users need to create or fix Redpanda Connect pipeline configurations. Trigger when users mention "config", "pipeline", "YAML", "create a config", "fix my config", "validate my pipeline", or describe a streaming pipeline need like "read from Kafka and write to S3".
Redpanda Connect Configuration Assistant
Create working, validated Redpanda Connect configurations from scratch or repair existing configurations that have issues.
This skill REQUIRES skills: component-search, bloblang-authoring.
Objective
Deliver a complete, valid YAML configuration that passes validation and meets the user's requirements.
Whether starting from a description or fixing a broken config, the result must be production-ready with properly secured credentials.
Handle Two Scenarios:
Creation - User provides description like "Read from Kafka on localhost:9092 topic 'events' to stdout"
Repair - User provides config file path and optional error context
This skill focuses ONLY on pipeline configuration orchestration and validation.
Skill Delegation:
NEVER directly use component-search or bloblang-authoring tools.
Component Discovery - ALWAYS delegate to component-search skill when it is unclear which components to use OR when you need component configuration details
Bloblang Development - ALWAYS delegate to bloblang-authoring skill when creating or fixing Bloblang transformations and NEVER write Bloblang yourself
Setup
This skill requires: rpk, rpk connect.
See the SETUP for installation instructions.
Tools
Scaffold Pipeline
Generates YAML configuration template from component expression.
Useful for quickly creating first pipeline draft.
Requires component expression specifying desired inputs, processors, and outputs
Expression format: inputs/processors/outputs separated by /
Multiple components of same type separated by ,
Outputs complete YAML configuration with specified components
--small flag omits advanced fields
Online Component Documentation
Use the component-search skill's Online Component Documentation tool to look up detailed configuration information for any Redpanda Connect component containing usage examples, field descriptions, and best practices.
Optional --env-file flag provides dotenv file for environment variable substitution
Optional --log.level DEBUG enables detailed logging for troubleshooting connection and processing issues
Starts pipeline and maintains active connections to inputs and outputs
Runs continuously until manually terminated with Ctrl+C (SIGINT)
Can be run repeatedly during pipeline development and iteration
Test with Standard Input/Output
Test pipeline logic with stdin/stdout before connecting to real systems.
Especially useful for validating routing logic, error handling, and transformations.
Example: Content-based routing
input:stdin: {}
pipeline:processors:-mapping:|
root = this
# Route based on message type
if this.type == "error" {
meta route = "dlq"
} else if this.priority == "high" {
meta route = "urgent"
} else {
meta route = "standard"
}
output:switch:cases:-check:'meta("route") == "dlq"'output:stdout: {}
processors:-mapping:'root = "DLQ: " + content().string()'-check:'meta("route") == "urgent"'output:stdout: {}
processors:-mapping:'root = "URGENT: " + content().string()'-check:'meta("route") == "standard"'output:stdout: {}
processors:-mapping:'root = "STANDARD: " + content().string()'
Test all routes:
echo'{"type":"error","msg":"failed"}' | rpk connect run test.yaml
# Output: DLQ: {"type":"error","msg":"failed"}echo'{"priority":"high","msg":"urgent"}' | rpk connect run test.yaml
# Output: URGENT: {"priority":"high","msg":"urgent"}echo'{"priority":"low","msg":"normal"}' | rpk connect run test.yaml
# Output: STANDARD: {"priority":"low","msg":"normal"}
Limitations:
Stdin/stdout cannot test batching behavior realistically
No connection, retry, or timeout logic validation
Cannot test ordering guarantees or parallel processing
Real integration testing still required before production deployment
YAML Configuration Structure
Top-level keys:
input - Data source (required): kafka_franz, http_server, stdin, aws_s3, etc
output - Data destination (required): kafka_franz, postgres, stdout, aws_s3, etc
