add-connector

name	add-connector
description	Add a new source/connector to devrel-ships (Luma, YouTube, Contentful, GitHub, etc.) so its "ships" flow into the DevRel Ships Notion DB. Use when the user wants to ingest a new source, implement/finish a connector stub, or asks "how do I add <source>". Covers the fetch function, the pure normalizer, host-identity extraction, the registry wire-up, and the test.

Adding a connector to devrel-ships

devrel-ships is a thin app on durable-sync: you write only the source-specific fetch + normalize. Upsert, dedup, roster matching, rate-limit handling, OAuth, worker registration, and scheduling all come from the library + ships_source.py / roster_enrich.py / pipeline.py. Read docs/integration-notes.md for the "why".

Steps

1. Decide access: pick the path that ships soonest WITHOUT waiting on another team. The deciding axis is which credential can you self-serve right now:

   • Direct first-party API when you can self-serve the key (Luma: mint it yourself).
   • OAuth-as-yourself via a trustworthy MCP server when the direct API's credential needs an admin/another team (Notion; Contentful w/o a delivery token). See [[mcp-vs-direct-api]] in docs/integration-notes.md §1.

2. connectors/<source>/client.py — the only place the source's HTTP details live. Paginate transparently. If the list endpoint omits data you need (Luma's list-events has no hosts), add a per-item enrichment call.

3. connectors/<source>/fetch.py — a plain fetch_<source>(after_iso: str | None) -> list[ShipInput] (NO @activity.defn — the library's fetch_source activity wraps it; sync/requests connectors are run in a thread by ShipsSource). Plus a pure normalizer (_to_ship-style: no IO, unit-testable). Map to ShipInput (see models.py). Key fields:

   • source, source_id (combined into the Sync Key dedup key — MUST be stable & non-empty), ship_name, ship_type (must match a Type select option name).
   • Host identities for matching: populate host_emails (preferred) AND host_names (fallback). roster_enrich tries email first. They ride the Record as transient props and are stripped before write.
   • status, url, ship_date_iso, reach, tags, source_author as available.

4. Register it: add ONE line to connectors/registry.py — "<Source>": fetch_<source> in CONNECTORS. ShipsSource turns it into a spec + entity workflow automatically; bootstrap.py starts it.

5. Test the normalizer in tests/ — pure, no network (see tests/test_normalize_luma.py). Most likely to drift when the source API changes.

6. Respect rate limits — handled for you: the library preloads the dedup map, reuses one MCP session, and backs off on 429; roster_enrich caches the roster. If your connector adds per-item enrichment, gate it behind a change-token check so unchanged past items aren't re-fetched every run.

Source-specific notes

• YouTube (stub): no per-video author field. Invert the search — scan title/description/tags for known roster names. Reach = viewCount.
• Contentful (implemented): scoped by content type, not per-author (config.CONTENTFUL_DEVREL_TYPES). introspect.py lists types/fields via the CDA.
• GitHub: decide "what counts as a ship" first (repo topic / ships.yml / Releases). NOTE: there's already a GitHub source in durable-sync — consider reusing durable_sync.sources.github instead of a from-scratch connector.

Verify

pip install -e ../durable-sync && pytest, then end-to-end: temporal server start-dev, library notion.bootstrap + notion.start (once), python -m devrel_ships.worker, python -m devrel_ships.bootstrap. A re-run should show created=0 (idempotent). Drive a single source with a sync_now signal on ships-sync:source:<Source>.