streamling

Use when implementing a streamling preprocessor (rewriting pipeline topology YAML before it parses), a DataFusion scalar UDF usable in SQL, a side output (observing every source without joining the pipeline), registering many component kinds in one crate, or dropping below the registration macros to the low-level manual FFI plugin module. Assumes streamling-plugin-basics.

Use when creating a new streamling plugin crate in Rust (source, sink, transform, preprocessor, UDF, or side output), or when wiring up registration macros, constructor contracts, the plugin lifecycle, the async runtime, error types, or option/secret handling. Start here before the type-specific skills.

Use when implementing a streamling SinkPlugin — lazy client initialization in initialize(), the empty-batch guard, converting RecordBatch to NDJSON for outbound writes, batched writes with retry and exponential backoff, partial-batch-failure handling that avoids silent data loss, and checkpoint-ack semantics for exactly-once. Assumes streamling-plugin-basics.

Use when implementing a streamling SourcePlugin — defining the output schema and _gs_op column, generate_batch and the empty-batch means-no-data contract, decomposing a source into fetcher/buffer/runner, resumable cursor or sequential pagination that survives restart via the checkpoint state backend, and output-channel backpressure. Assumes streamling-plugin-basics.

Use when implementing a streamling TransformPlugin — process_batch with RecordBatch in and RecordBatch out, declaring output_schema, filtering or projecting with Arrow compute (boolean masks, filter_record_batch), handling empty batches, and passing checkpoints through. Use this instead of the built-in sql transform when row logic needs custom Rust or external calls. Assumes streamling-plugin-basics.

Use when implementing a streamling DataFusion scalar UDF — a custom function callable from pipeline SQL (SELECT my_func(col) FROM ...). Covers the ScalarUDFImpl trait, the modern invoke_with_args / ScalarFunctionArgs API, argument validation and downcasting, null propagation, return_type vs return_field_from_args, volatility, both registration macros, and the create_udf legacy path. Assumes streamling-plugin-basics.