real-time-sources

Use when adding AMQP 1.0 transport to a feeder in this repo — either alongside an existing Kafka/MQTT feeder or for a brand-new source. Covers generic AMQP brokers (RabbitMQ AMQP 1.0 plugin, ActiveMQ Artemis, Qpid Dispatch) via SASL PLAIN, Azure Service Bus / Event Hubs via Entra ID (AMQP CBS put-token, no SAS-key minting), xRegistry AMQP endpoint and messagegroup pattern, the `--azure_cbs_target` xrcg generator switch, two-app folder layout sharing a core acquisition module, Dockerfile.amqp, ARM template for Service Bus, and CI matrix integration. Pegelonline is the reference implementation.

Use when packaging or finishing a source for merge in this repo. Covers Dockerfile patterns, CONTAINER.md authoring, Azure Container Instance templates, mandatory KQL schema script generation + KQL Optimizer review, optional Fabric assets, EVENTS.md generation, and repo-level delivery checks including Docker E2E validation.

Use when performing end-to-end deployment validation of sources in this repo. Validates that data flows from upstream APIs through containers/notebooks into Event Hubs (Azure) or KQL databases (Fabric), and that Maps/Dashboards render live data. Covers per-source Azure ACI validation, Fabric notebook validation, KQL row verification, Map layer data checks, issue filing, and cleanup.

Use when adding MQTT/Unified-Namespace transport to a feeder in this repo — either alongside an existing Kafka feeder (transport split) or for a brand-new source where MQTT is the primary fit. Covers UNS topic-tree design, xRegistry MQTT messagegroup pattern, paho-mqtt v5 binary-mode CloudEvent emission, Entra JWT enhanced auth for Event Grid namespace brokers, two-app folder layout sharing a core acquisition module, Dockerfile.mqtt + dedicated pyproject, two MQTT ARM templates (BYO broker + Event Grid namespace), the read-only test-client registration helper, MQTT Docker E2E, and the four-button portal/README update. NOTE: AMQP 1.0 transport is MANDATORY alongside MQTT — see `.github/skills/amqp-feeder/SKILL.md` and the 'Mandatory AMQP Companion' section below.

Use when implementing the runtime bridge (feeder) for a source in this repo. Covers pollers, websocket or MQTT clients, raw TCP decoders, family-aware normalization, generated producer wiring, state and dedupe logic, source-local testing, mandatory KQL schema generation and KQL Optimizer review, mandatory expert review of the contract and generated schemas, and the Fabric notebook hosting option for poll-based sources (both new sources and retrofits of already-shipped sources).

Use when planning or adding a new real-time source in this repo. Covers mandatory upstream docs and field review, required JSON Structure Core and extension-spec study for schema work, repo fit, transport choice, event-family scoping, folder layout, and sequencing xreg, runtime, container, and documentation work.

Use this checklist BEFORE merging any feeder change in this repo (new source, new transport variant, contract change, schema change, env-var change, deploy template change, doc change). Covers schema description quality, transport completeness (Kafka + MQTT + AMQP), ghpages portal deploy buttons, README + CONTAINER.md business-value framing, env-var documentation, ARM templates, EVENTS.md regen, Docker E2E coverage. EVERY item must be ticked or explicitly justified as not-applicable before the PR is shippable.

Use when adding or modifying a Fabric notebook-hosted feeder for a source in this repo. Covers per-source Fabric Environment build (poetry wheel + path-dep stripping), CustomEndpoint connection-string lookup via the public Topology API, Lakehouse + KQL + Environment binding, scheduling, and OneLake-based diagnostics for scheduled runs.

Use when designing or changing event contracts for this repo. Covers mandatory upstream schema review, required JSON Structure Core and extension-spec study, exhaustive field descriptions from upstream docs, CloudEvents modeling, xreg message groups, stable subject and Kafka key selection, JsonStructure extension usage, regeneration with xrcg 0.10.8, and EVENTS.md refresh.

Use when searching for new real-time or near-real-time open data sources worldwide that could be added to this repo. Covers domain taxonomy, search strategies, protocol fitness, freshness probing, and candidate ranking.