live-connectors

name

live-connectors

description

Wire a live GL/accounting-system OAuth2 extractor (QuickBooks Online / NetSuite / Sage Intacct / Xero) that feeds the canonical trial-balance staging seam, using the reference-implementation token client in scripts/connectors/ — atomic persist-then-use rotating refresh, per-entity lock, error-cause routing (401 refresh / 429 backoff / invalid_grant re-auth), Xero 30-min grace — plus drill-through GL lineage that feeds statement_engine --gl-detail unchanged. Reference impl + record/replay harness; NOT live-verified. Used by `controller`.

Skill: live-connectors

Purpose: turn "a controller has OAuth access to their GL" into "a raw export lands in the canonical staging seam, with drill-through provenance" — safely. The single highest-consequence failure in this tier is a dropped rotating refresh token (QBO + Xero): lose it on a crash and you are locked out mid-close until a human re-auths. This skill's reference implementation encodes the four disciplines that prevent that, and a record/replay harness so you can develop and test the whole path with zero live credentials and zero network.

HONEST BOUNDARY — read first. Everything here is a reference implementation + offline harness, NOT a live-verified, certified connector. No live credentials are used; no live socket is ever opened by this code. Every provider "fact" in this skill and in ../../knowledge/finance-elt-connector-facts.md — token lifetimes, rate limits, the QBO/Xero rotating-refresh behavior, the Xero 30-minute grace window, error-code semantics — is doc-sourced, not live-observed; re-confirm before go-live. It is training knowledge, settling-gated: browser-verify each against its primary provider doc before it gates a live build; do not treat any of it as observed from a running integration. Live wiring — the real IdP, the real client credentials, the real token/report endpoints, and the real warehouse to land into — is the consumer's step. Outputs are decision-support scaffolding, not an accounting/audit/tax opinion and not a competitive claim of a working integration (see ../../CLAUDE.md §3).

What's in the box

File	Role
../../scripts/connectors/oauth_client.py	Token-lifecycle client. Providers-as-data; atomic persist-then-use rotating refresh; per-entity fcntl lock; error-cause routing; Xero 30-min grace. security_review target — token handling.
../../scripts/connectors/replay_transport.py	Stdlib record/replay transport. Serves recorded synthetic fixtures; fails loudly on a missing fixture; never opens a socket.
../../scripts/connectors/adapters/	Per-provider adapters (QBO / NetSuite / Sage Intacct / Xero). Each emits a raw export file that tb_stage.py already normalizes — staging is not re-implemented.
../../scripts/connectors/gl_lineage.py	Drill-through lineage: first 6 columns byte-identical to statement_engine --gl-detail, plus source_system,source_type,source_id,source_doc_url. Feeds the engine unchanged.
../../scripts/connectors/fixtures/	Synthetic, obviously-fake recorded fixtures (fake realm/tenant ids; fake token values) per provider.
../../scripts/test_connectors.py	31-test acceptance suite (all the invariants below).

The rotating-refresh failure mode and the four disciplines

For QBO and Xero the refresh token rotates: a successful refresh returns a NEW refresh token and kills the old one. The client encodes all four required mitigations:

1. Persist-then-use, atomically. The new token pair is written to a temp file and os.replace()d into the store before the new access token is ever used. Consequence: a crash after the rename leaves the NEW token durable; a crash during the write leaves the OLD token fully intact. There is never a half-written store and never a silent lockout.
2. Per-entity exclusive lock. An fcntl.flock on a per-entity lockfile wraps the read-refresh-write critical section; a recheck-under-lock collapses two racing refreshes to one rotation (two processes can't each rotate and invalidate the other).
3. Error-cause routing — the cause selects the fix, and they are not interchangeable. 401 (expired access) → refresh + retry the same route; 429 → backoff honoring Retry-After; 400 invalid_grant (dead refresh) → REAUTH_REQUIRED, non-retryable, fire the alert hook, never backoff. Guessing the cause picks the wrong fix.
4. Xero 30-min grace. A refresh whose response is lost leaves you unsure whether the server rotated; within the provider grace window the client retries with the existing refresh token rather than assuming a rotation it never observed.

Consumer OAuth wiring runbook

1. Copy the config template per entity — ../../templates/connector-config.template.json. Set source_system, the one matching source_ids.*, the token_store_path, and — critically — set rotating_refresh_token: true for QBO and Xero.
2. Provision credentials as ENV-VAR NAMES only. The template stores client_id_env, client_secret_env, refresh_token_env — the names of environment variables, never the values. Populate the actual secrets in your runtime's secret manager. Never commit a token value or a .token.json store.
3. First-time auth (interactive). Run the provider's authorization-code flow (Xero: + PKCE) to obtain the first access+refresh pair, and write it to token_store_path (0600). This is the one step that needs a human/browser; automate the rest.
4. Implement the live transport. oauth_client.OAuthClient takes an injected transport with a token_request(url, data) -> (status, body, headers) seam. For development/tests use ReplayTransport; for production supply a real HTTPS transport (e.g. stdlib urllib/http.client or your HTTP library) that calls the provider token_url. The token-lifecycle disciplines live in OAuthClient, so the live transport stays thin.
5. Wire the alert hook + re-auth runbook. Pass alert_hook= so an invalid_grant pages an on-call controller. Document the interactive re-consent path per source (authorize URL → consent → capture code → exchange → persist atomically) so access is restorable without an engineer.
6. Extract → adapter → stage. Pull the TB/report, hand it to the provider adapter to emit a raw export file, then normalize with tb_stage.py stage using that provider's column-map. The staged CSV flows straight into statement_engine.py --tb.
7. (Optional) drill-through lineage. Build a lineage file with gl_lineage.py build (a --gl-detail file + a source-doc sidecar) and pass it to ; the badge lifts to and the source-doc keys ride into the reasoning trail.

# Offline (replay) end-to-end, zero credentials:
python3 scripts/connectors/gl_lineage.py build \
  --gl-detail skills/produce-gaap-statements/examples/gl-detail-2026-06.csv \
  --source-system qbo \
  --docs scripts/connectors/fixtures/lineage/source-docs.csv \
  --out /tmp/lineage.csv
python3 scripts/statement_engine.py --entity <entity.json> --coa <coa.csv> \
  --tb <staging.csv> --gl-detail /tmp/lineage.csv   # badge -> GL-detail-traced

Verify before you trust the numbers

Run the acceptance suite: python3 scripts/test_connectors.py (31 tests, stdlib-only). It proves the persist-then-use ordering, crash-safety both ways, the one-rotation lock, the invalid_grant re-auth path, the three-way error routing, the Xero grace retry, the replay transport's loud missing-fixture failure + no-socket guarantee, and the lineage byte-identity + badge lift. Before a live build, also browser-verify the settling-gated provider facts in ../../knowledge/finance-elt-connector-facts.md.

What this is not

Not a certified or live-verified connector; not an accounting/audit/tax opinion. The harness proves the token-handling and lineage disciplines against synthetic fixtures — it does not prove the source GL is correct, reconciled, or complete. A clean extract is a necessary, not sufficient, condition for a trustworthy close: reconciliation (../reconciliation-summary/SKILL.md / ../reconciliation-automatch/SKILL.md) and the governed review→approve→lock spine (../close-approval-workflow/SKILL.md) still apply.