Teaches how to build streaming data pipelines with Redpanda Connect (formerly Benthos) — declarative YAML config, the input/pipeline/output model, Bloblang mapping language, error handling, and canonical pipeline patterns. Use when: building or running a Redpanda Connect pipeline; wiring an input to an output with processors; writing Bloblang mappings or mutations; choosing connectors (kafka, redpanda, http_server, file, generate, sql_*, aws_*, gcp_*); running pipelines with rpk connect run or redpanda-connect; configuring buffers, caches, metrics, rate limits, or the global redpanda block; writing config.yaml for redpanda-connect; asking what Connect components exist and how to discover them; error handling with fallback outputs or catch processors; batching records before writing; understanding the difference between mapping and mutation processors; choosing between the bloblang and mapping processor names; connecting to Kafka or Redpanda with SASL/TLS from Connect; migrating Kafka clusters to Redpanda with
2026-07-10
مديرو الشبكات وأنظمة الحاسوب Provision and manage Redpanda Cloud BYOC (Bring Your Own Cloud) clusters via the Control Plane API (https://api.redpanda.com) and the rpk cloud byoc agent plugin. Covers OAuth2 client-credentials auth, creating Network resources per cloud provider (AWS VPC/subnets/IAM, GCP network/project/bucket, Azure VNet/subnets/identities), creating TYPE_BYOC clusters (network_id, throughput_tier, cloud_provider_tags, customer_managed_resources), polling long-running Operations until READY, and running the Terraform-backed rpk cloud byoc apply/destroy/validate flow with --redpanda-id. Use when: creating BYOC clusters in AWS/GCP/Azure via the public API; provisioning or tearing down BYOC networks; wiring customer-managed IAM roles/buckets/subnets into a Redpanda cluster; setting up private connectivity (AWS PrivateLink, GCP Private Service Connect, Azure Private Link, VPC/network peering via NetworkPeeringService, and AWS Transit Gateway centralized egress); registering cross-account AWS access (CloudProviderAccessService,
2026-07-09
مديرو الشبكات وأنظمة الحاسوب Configures and operates an Oxla cluster: YAML config (default_config.yml), OXLA__ environment-variable overrides, node ports and roles, leader election, storage backends (local/S3/GCS/Azure), memory limits, access control, TLS, OIDC, and the HTTP-based ConnectRPC admin service (port 9090) with its LoggingService (GetLogLevel/SetLogLevel). Also covers the Prometheus metrics endpoint (port 8080) and Docker/Compose deployment patterns. Covers Oxla's lakehouse and streaming differentiators: Apache Iceberg REST catalogs (CREATE ICEBERG CATALOG, oauth2/basic/aws_sigv4 auth, feature_flags.allow_iceberg_queries), transparent Redpanda/Kafka integration (CREATE REDPANDA CATALOG, topic-backed tables, schema_lookup_policy/error_handling_policy), object-storage connections (CREATE STORAGE TYPE S3/GCS/ABS), plus security: OIDC/JWT auth, SCRAM passwords, centralized access control, and AES-256-GCM secret encryption (OXLA_ENCRYPTION_KEY). Use when: configuring an Oxla cluster; setting ports or node names; enabling TLS or OID
2026-07-08
Query external data from Oxla — Kafka topics via catalogs, Apache Iceberg tables, and S3/GCS/Azure parquet/ORC files — alongside native Oxla tables. Use when: querying Kafka topics with CREATE KAFKA CATALOG or CREATE REDPANDA CATALOG; reading Apache Iceberg tables using the catalog=>path.table syntax; loading or exporting parquet/ORC files from S3/GCS/Azure (COPY FROM/TO); joining external data with native tables; or inspecting external metadata via system.kafka_connections, system.kafka_sources, system.iceberg_catalogs, and system.iceberg_tables. Also covers the Redpanda Enterprise features that produce the data Oxla reads — Iceberg Topics (redpanda.iceberg.mode and the redpanda.iceberg.* topic properties; iceberg_enabled and iceberg_rest_catalog_* cluster properties), Tiered Storage (cloud_storage_enabled), and Server-Side Schema ID Validation (enable_schema_id_validation, redpanda.value.schema.id.validation) — all of which require a Redpanda Enterprise license. Trigger phrases: "query Kafka topic from Oxla
2026-07-08
Write and run SQL against Oxla, a PostgreSQL-wire-compatible distributed columnar analytical database. Covers connecting via psql or any PostgreSQL driver (port 5432, password "oxla" by default), the full supported data-type set (INT/LONG/INT16/INT32/FLOAT/DOUBLE/CHAR/VARCHAR/STRING/TEXT/DATE/TIME/ TIMESTAMP/TIMESTAMPTZ/INTERVAL/BOOL/JSON/JSONB/BYTEA/NUMERIC/ARRAY/GEOMETRY/ GEOGRAPHY/POINT), DDL (CREATE/DROP TABLE, CREATE TABLE AS SELECT, CREATE/DROP VIEW, CREATE/DROP SCHEMA, TRUNCATE, CREATE ROLE/GRANT/REVOKE), DML (SELECT/INSERT/UPDATE/DELETE, SELECT INTO table or file), data loading (COPY FROM/TO with CSV/Parquet/ORC formats, S3 credentials), aggregate and window functions (SUM/AVG/COUNT/MIN/MAX/percentile_disc/percentile_cont/mode, ROW_NUMBER/RANK/DENSE_RANK/LAG/LEAD/FIRST_VALUE/LAST_VALUE/NTH_VALUE/NTILE/ CUME_DIST), CTEs (WITH), UNION/INTERSECT/EXCEPT (with optional ALL), PREPARE/EXECUTE, and analytic query patterns (GROUP BY, ORDER BY, LIMIT/OFFSET, multi-table joins, star-schema aggregations). Also co
2026-07-08
مديرو الشبكات وأنظمة الحاسوب Collects local and remote Redpanda diagnostics bundles and gathers broker process info using the `rpk debug` command group. Use when: collecting a debug bundle for Redpanda support or self-triage, generating a remote (cluster-wide) bundle via the Admin API, gathering local broker process or diagnostic info from the CLI, using rpk debug bundle or rpk debug remote-bundle, troubleshooting a Redpanda cluster and needing to capture logs/metrics/profiles, passing a debug bundle to the support team, or running rpk debug on Linux or Kubernetes. Also covers triaging enterprise (license-gated) features from a bundle — Tiered Storage, Cloud Topics, Iceberg Topics, Continuous Data Balancing, Shadow Linking (cross-cluster DR), Remote Read Replicas, Audit Logging, RBAC/GBAC, OIDC/OAuthBearer/ Kerberos auth, FIPS mode, Server-side Schema ID Validation, Schema Registry Authorization, and Leadership Pinning, plus checking license status and license violations. Applies to self-managed deployments only (Linux hosts and Kubernet
2026-07-07
مديرو الشبكات وأنظمة الحاسوب Manage Redpanda topics and produce/consume records from the CLI using the `rpk topic` command group. Covers creating topics with custom partition counts, replication factors, and configs; listing and describing topics (summary, configs, partitions); altering topic configs (set/delete/append/ subtract); adding partitions; trimming/deleting records; analyzing throughput and batch size; describing tiered-storage status; and deleting topics. Also covers `rpk topic produce` and `rpk topic consume` in depth including the --format percent-escape syntax, keys, headers, compression, acks, tombstones, schema-registry encoding/decoding, and all --offset forms. Also covers Redpanda enterprise topic properties (Enterprise license required): Tiered Storage (redpanda.remote.read/write/delete/recovery, redpanda.storage.mode, retention.local.target.*), Cloud Topics (redpanda.cloud_topic.enabled / redpanda.storage.mode=cloud), Iceberg Topics (redpanda.iceberg.mode/delete/partition.spec/target.lag.ms/ invalid.record.action), Re
2026-07-07
Builds, deploys, and manages Redpanda Data Transforms — WebAssembly (Wasm) functions that run inside the broker to transform records in-flight from an input topic to one or more output topics. Covers the full lifecycle: enabling the feature, initializing a project, writing transform logic in Go (TinyGo), Rust, JavaScript, or TypeScript, building the Wasm binary, deploying with rpk transform deploy, listing and inspecting running transforms, viewing logs, pausing/resuming, and deleting transforms. Use when: writing or deploying a Redpanda data transform, in-broker Wasm transform, wiring an input topic to an output topic, filtering or transforming Kafka records inside the broker, using rpk transform init / build / deploy / list / logs / pause / resume / delete, enabling data_transforms_enabled cluster config, using the Go transform SDK, Rust transform SDK, or JavaScript/TypeScript transform SDK, from-offset reprocessing, or setting environment variables on a transform. Also covers wiring transform output topics
2026-07-07