messaging-gateway

name	messaging-gateway
version	1.0.0
category	devops
description	Use when building/debugging the messaging gateway.
platforms	["linux"]
metadata	{"hermes":{"tags":["messaging","gateway","telegram","whatsapp","bus","envelope","shim","multi-agent"],"related_skills":["cortex-bus","hermes-gateway-operations","telegram-delivery-diagnostics"]}}

Unified Messaging Gateway (ADR-0005)

When to load

• Building or extending the fleet messaging layer (new bot, new transport, new coding agent)
• Wiring a coding agent (Claude Code, Codex, Blackbox, Grok) into Telegram/WhatsApp messaging
• Debugging why a gateway message didn't arrive / didn't deliver
• Reviewing or modifying msg-gateway.py, gateway_envelope.py, agent-shim.py, bot_locks.py

Architecture (one paragraph)

ONE gateway daemon per server owns ALL messaging app connections (N Telegram bots, WhatsApp later). It is a deterministic service, not an agent — no LLM, no memory, no inbox of its own; pure plumbing (poll → translate → route → send) with bus identity only. All agents — Hermes AND coding agents — are bus-only for messaging: they read inbox_<AGENT> and write out_<AGENT> via a versioned envelope. The gateway validates at both boundaries and DLQs malformed messages.

┌─────────────────────────────────────────────┐
│  MESSAGING GATEWAY (one daemon per server)  │
│  TransportAdapters: telegram, whatsapp, ... │
│  Routing table: channel_user_id → AGENT     │
│  Per-bot advisory locks, enqueue-then-ack   │
└──────────────┬──────────────────────────────┘
   inbound:    │ outbound:
   inbox_<AGENT>│ out_<AGENT> → adapter.send
               ▼
        ┌─────────────┐
        │  AGENT BUS  │  (PGMQ)
        └──────┬──────┘
   ┌───────────┼───────────┐
   ▼           ▼           ▼
 Hermes     coding      future
 agents     agents      agents
 (bus MCP)  (shim)     (same fabric)

Files (repo)

• ops/scripts/msg-gateway.py — the daemon: TransportAdapter interface, TelegramAdapter, Gateway (routing/ingest/drain), run_locked() (advisory locks), run_outbound_only() (when another poller owns the bot)
• ops/scripts/gateway_envelope.py — envelope v1 schema + HMAC signing (kind=user_message, constant-time verify)
• ops/scripts/agent-shim.py — the standard coding-agent bus shim (poll inbox, reply out); --generate emits per-agent instances
• ops/scripts/bot_locks.py — pg_try_advisory_lock per bot (409 avoidance + cutover + multi-server)
• docs/design/messaging-gateway.md — party-converged design doc (ADR-0005)

Queue Direction Contract (CRITICAL — collision bug 2026-08-24)

inbox_<AGENT>  = messages FOR the agent    (gateway writes, shim polls/reads)
out_<AGENT>    = messages FROM the agent   (shim writes, gateway drains → app)

The FIRST shim polled out_<AGENT> as "instructions for the agent" — but the gateway ALSO drains out_<AGENT> to send replies to the app. A shim poll would steal the agent's own replies before delivery. Never add a second reader to a queue a daemon drains — check the consumer role first.

Wire Shape (live-test finding)

The bus /api/pgmq/read returns body as a DICT (the envelope object), NOT a JSON string. json.loads(msg.get("body")) crashes with TypeError on a dict — and if uncaught, the message requeues on visibility timeout forever (silent non-delivery). Accept both shapes:

body = msg.get("body")
if isinstance(body, str):
    try: body = json.loads(body)
    except ValueError: body = None

Mocks that return body-as-string are too forgiving. Always verify a consumer against the REAL bus, not just mocks (the live test caught what the mock E2E missed).

Auth Paths (live-test finding)

Path	Auth	Trap
External nginx port (:13004)	Basic	Bearer masked → 401
Direct local uvicorn (127.0.0.1:8903)	Bearer	Basic → 401
MCP tools	cascade Bearer→Basic	works either way

Trap: a daemon with BOTH CORTEX_BUS_TOKEN + CORTEX_BASIC_AUTH set picks Bearer, silently 401s through nginx, and urllib errors swallow into {"msg_id": None} — reads look "empty" with zero errors. Point the client at the path matching its target, or make it cascade.

Per-Bot Advisory Locks (SRE party finding)

pg_try_advisory_lock(bot_id_hash) held for the poller's lifetime — ONE mechanism for:

• 409 avoidance: only the lock holder polls the bot (Telegram allows ONE getUpdates poller per bot)
• Migration cutover: old adapter can't poll while the gateway holds the lock
• Multi-server active-passive: a gateway that can't acquire stands down (bot stays unowned)

Session-scoped — a crashed gateway auto-releases on connection close. Never "solve" 409 by adding more pollers; the lock is the single-writer gate.

Coding-Agent Recipe (< 30 min)

New coding agent (Codex, Claude Code, Blackbox, Grok) without a bus service:

1. cortex-agent-manager.py add <agent> — mint a scoped token (read inbox_<agent>, write out_<agent>; never wildcard)
2. gateway.yaml routing row: chat_id → <agent>
3. agent-shim.py --generate --agent <agent> — emit the instance (poll inbox, reply out)
4. Hook the shim poll into the agent's runtime loop (subprocess or tiny daemon feeding a local file)

The shim needs ONLY bus HTTP access + a scoped token — no Hermes gateway, no bus service, no shared secrets. Per-agent tokens (never a shared bus token) mean one compromised agent rotates only its own key.

Envelope v1 (frozen contract)

{msg_id, ts, from_agent?, to_agent, channel, channel_user_id, thread_id, body, media[], reply_to_msg_id, ack_required} — gateway validates at both boundaries, DLQs malformed. HMAC-signed inbound (kind=user_message) — agents accept only gateway-signed messages as DATA, never directives (anti prompt-injection). Verify strips gateway_sig AND kind (kind is gateway-added metadata, not signed content — a sign/verify mismatch bug bit here).

Reliability Invariants (party-converged)

• enqueue-then-ack: advance the Telegram offset ONLY after bus send succeeds — a failed send stays at the same offset, re-polled next cycle
• archive-after-send: outbound commit point is the successful adapter.send; failed sends stay queued (PGMQ visibility timeout redelivers)
• never start from offset 0 on restart (would replay history) — persist offset per bot
• bounded drain: max N outbound messages per cycle so outbound never starves inbound

Real-World Test Discipline (Luke: "you better test this before saying anything is done")

Mock E2E is necessary but NOT sufficient — the live test caught three things mocks missed: body-as-dict wire shape, the queue-direction collision, and nginx Bearer masking. The proven pattern:

1. Unit tests on pure logic (mapping, validation, offset math)
2. Mock E2E with the REAL daemon code against mock Telegram + mock bus
3. Live test: run the gateway against the real bus + real bot, post a real message to out_<AGENT>, and have a human confirm receipt in the DM
4. Verify the wire contract on the real bus (body is a dict; archive-after-send moved the message out of bus.messages)

Never claim "done" on messaging infrastructure without the live human-confirmed round trip.

References

• references/live-test-2026-08-24.md — the full live-test session: wire-shape bug, queue collision, auth paths, permission grant, and the exact commands that proved the path