Use when verifying Relayfile provider integrations end-to-end against a real workspace, including auth, OAuth connect, sync, file-tree shape, discovery materialization, outbound writeback, inbound webhook checks, evidence capture, and issue filing.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Use when verifying Relayfile provider integrations end-to-end against a real workspace, including auth, OAuth connect, sync, file-tree shape, discovery materialization, outbound writeback, inbound webhook checks, evidence capture, and issue filing.
Integration Verification Playbook
The canonical methodology for proving — with empirical evidence against a real Relayfile workspace — that a provider integration actually works end-to-end. Read this in full at the start of every verification task. Update it whenever a new failure mode appears.
Why this playbook exists
A Nango integration shipping does not equal an integration that works. Across the May 2026 verification effort, the following gaps were only visible by actually exercising the end-to-end flow on a real workspace, never by reading PR descriptions:
Discovery schemas materializing at the advertised paths but containing zero properties for some flat resources (cloud#778).
A control-plane endpoint silently 15–90s slow under realistic load, so the CLI's HTTP client times out before the cloud responds (cloud#766).
Sync workers silently no-op'ing because of a plan/quota gate, while reporting backfilled:true.
A Worker bundle transitively pulling pg from a sibling import, breaking deploys (cloud#762).
Each of these passed CI. Each was caught only by running the full flow. The playbook locks in that the full flow gets run.
The seven phases
For every provider × workspace, run these in order. Don't skip phases. If a phase is honestly skippable (e.g. provider does not support outbound writes), note that explicitly with the reason — never silently.
Phase 1 — Auth
Verify the workspace token in is valid:
Expected: .
If 401: refresh. Try relayfile login --no-open first. If it emits warning: could not refresh workspace token for <ws>: ... context deadline exceeded (the cloud#766 symptom), mint the token directly with curl:
If not connected: relayfile integration connect <p> --workspace <ws> --no-open and complete OAuth in the browser. The CLI poll resolves once the cloud reports oauth.connected:true (often <5s after the Nango webhook fires; see cloud#736 for the connection-created ingress fix that made this reliable).
Red flag (file an issue if seen):oauth.connected:false with no error after completing OAuth, OR connectionMatched:false with a non-null currentConnectionId — that's a Nango webhook ingestion gap (the cloud#736 family).
Phase 3 — Sync
Wait for initialSync.state === "complete". If it gets stuck in queued with syncName:null / model:null, that's typically a Nango plan/quota issue — the sync worker accepted the job but Nango never registered the actual sync. Confirm with the user (the operator can check Nango's billing/usage UI) before assuming it's a code bug.
Spot-check the actual canonical record path (e.g. /<p>/<resource>/<canonical-filename>.json) and the by-id/by-uuid/by-* alias subtrees the LAYOUT advertises. They must actually exist.
If LAYOUT advertises discovery/<p>/.../.schema.json but no /discovery/<p>/ subtree exists, that's the pre-#761 condition and is itself a finding.
Expected: 200 with body { "workspaceId": "<ws>", "refreshed": [{ "provider": "<p>", "discoveryBackfilled": true, "errors": 0 }, ...] }.
Note: this endpoint is currently slow (commonly 90–180s for 6 providers). 240s timeout is a safe floor.
Read each writable resource's discovery files:
for path in \
"/discovery/<p>/.adapter.md" \
"/discovery/<p>/<resource>/.schema.json" \
"/discovery/<p>/<resource>/.create.example.json"; do
curl -sG "https://api.relayfile.dev/v1/workspaces/<ws>/fs/file" \
--data-urlencode "path=$path" -H "Authorization: Bearer $TOKEN"
done
For .schema.json: parse content, report properties key count + lastEditedAt.
The cloud#778 pattern to watch for:refreshed[].discoveryBackfilled === true but properties: {} AND lastEditedAt is older than the refresh timestamp. That means writeManagedFile dedup'd a byte-identical empty schema — the sampling silently produced zero records for that resource. File a separate issue per provider × resource with the empty schema, citing the stale lastEditedAt as evidence.
The cloud#761 honest scope: placeholder-path resources (/<p>/<parent>/{placeholder}/<sub>) are intentionally NOT backfilled by #761 — their schemas stay permissive-empty by design. Note these explicitly; do not confuse with #778.
Phase 6 — Outbound writeback
Only run if Phase 5 produced a non-empty .schema.json for at least one resource AND a safe target exists.
Target selection
In order of preference:
Schema-validated + pre-existing throwaway: a record explicitly marked relayfile-writeback-test-* (e.g. KAN-4 in rw_fc7b534b).
Schema-validated + dedicated test record the operator creates upfront and confirms is throwaway.
LAYOUT-documented edit on a confirmed throwaway: when no schema exists but the LAYOUT documents the contract.
Skip: anything else. Do not write to a real production record to "test the chain."
Construct the mutated content: read the schema, identify a mutable (readOnly:false) field, modify it with the marker. Marker format: [relayfile writeback test <ISO-utc>]. For dict-shaped fields (e.g. Jira ADF descriptions), append a structured node. For string fields, append the marker text. Preserve all other fields verbatim.
Expected: HTTP 202 with {"opId":"op_<n>", "status":"queued", "targetRevision":"<new-rev>", "writeback":{"provider":"<p>","state":"pending"}}.
Round-trip verification
Wait 30–60s for the writeback executor to PATCH the provider.
Read the op state: GET /v1/workspaces/<ws>/ops/op_<n>. Expected: status: "succeeded" or "completed". Watch for the cloud#780 pattern: status:"failed", attemptCount:0, lastError:"No <p> writeback rule matched <path>" — the routing layer rejected the path because the adapter has no edit rule. File an issue per the template.
Re-read the canonical record from the workspace. Expected: new revision, marker present in the mutated field.
(Optional but recommended) Verify in the provider's actual UI/API that the change landed (e.g. the Jira REST API for the issue). If the cloud-side record shows the marker but the provider doesn't, the writeback succeeded locally but the outbound failed — that's the same class as cloud#780.
Create flow (non-canonical filename)
Use <resource>/.create.example.json as the starting point. Customize the marker in a clearly-labeled field (e.g. the issue/comment/page title or body).
PUT to a non-canonical filename inside the resource directory (e.g. /<p>/<resource>/wb-test-<utc>.json).
Expected: adapter creates the real provider record, the canonical <id>.json appears with the synced data, and the original draft file is rewritten as a pointer/receipt referencing the canonical path. Verify both the canonical record exists AND the draft was rewritten (not just left as-is, not deleted).
The pointer-rewrite is the documented file-native create contract; if the draft stays as a raw payload or vanishes without a receipt, that's a finding.
Phase 7 — Inbound (webhook-driven sync)
This phase requires the operator to make a change in the external provider UI/API. Coordinate with them.
Ask the operator to make a small, identifiable change in the provider (e.g. add a label to a Linear issue, edit a Notion page title, append text to a Confluence page, send a test message to a designated Slack channel). Note the timestamp.
Within ~60s, the Nango webhook should fire → cloud → file event → workspace tree update.
Re-read the affected canonical record from the workspace. Expected: the change is reflected in the file content, the file's revision advanced.
Also check /digests/today.md — the change should appear in the activity summary for the provider (assuming the digest internal-path filter from cloud#771 is correctly excluding internals).
If the change does not propagate within ~5 minutes: file a webhook-ingress issue with the operator's exact action, timestamp, expected vs observed.
Recurring failure patterns
When you see these, file the issue immediately — they are known classes with known evidence:
Pattern
Looks like
Filed as
Issue template focus
Empty discovery schema masked by dedup
backfilled:true but properties:{} AND lastEditedAt older than refresh
cloud#778
Resource × root cause hypothesis (row-id vs alias-key mismatch); add behavioral regression test asking
Cite both the LAYOUT contract AND the live emit path; ask for the rule + a regression test
Pathologically slow control-plane endpoint
Same endpoint, multiple HTTP timings, variance ≥10×
cloud#766
Always include a comparison to fast endpoints (/api/health, /fs/tree) at the same time
Worker bundle drift
B1 worker import safety fails with Node builtin resolution errors
cloud#762
The fix is import-graph split, NEVER an external allowlist
Connection-created ingestion gap
oauth.connected stays false after a successful Nango OAuth
cloud#736
Check connect-session pre-creates a pending row; webhook ingress; mock vs prod path divergence
Worker dedup at parity-enablement
(latent) Worker queue consumer doing real work without persistent dedup
cloud#775
At-least-once → duplicate side effects; needs D1 or postgres-js-over-Hyperdrive
Issue-filing template
Every gap discovered during verification becomes a GitHub issue. Use this template:
## Summary
[1–3 sentences: what happens, where, and why it matters operationally]
## Empirical evidence
[Curl response excerpts, file-content excerpts, file paths + revisions. Concrete, not summarized.]
## Repro
[Numbered steps an engineer can copy-paste. Include exact paths, exact headers if relevant, the workspace id if it's a published one.]
## Suspected root cause
[Hypothesis with file:line refs if available. Be explicit that it's a hypothesis vs verified.]
## Why it matters
[Operational impact — does it block a documented feature? Mislead an agent? Silently corrupt data?]
## Asks / next steps
[Concrete, numbered. E.g. "Trace X", "Add regression test for Y at file:line", "Update the operator-facing doc Z".]
## Context
[Cross-link related issues + PRs. Mention which verification phase surfaced it.]
Test-marker conventions
Every outward-facing write (writeback test, create test) MUST include a clearly-marked machine-greppable marker so the operator can find + clean up afterwards:
Format: [relayfile writeback test <ISO-utc>] — e.g. [relayfile writeback test 2026-05-20T08:09:56Z].
For dict-shaped fields (ADF, JSON-LD bodies): embed the marker as a structured paragraph/text node so it doesn't break the document.
For string fields: append on a new line with a leading blank line.
For create operations: include the marker in the title/name AND the body, so it's discoverable from both the listing view and the detail view.
After every writeback test, output: the provider, the exact record id, the path in the workspace, the timestamp marker. That output is what the operator uses to manually verify + clean up.
Verification matrix output
Final report MUST include a per-provider × phase matrix:
Provider
Auth
Connect
Sync
Tree
Discovery
Writeback-out
Webhook-in
gmail
✅ HTTP 200
✅ oauth.connected:true
✅ initialSync.state:complete ( records)
✅ matches LAYOUT
⚠️ schema empty (cloud#NNN)
⏭️ skip (no schema)
❌ change not propagated (cloud#MMM)
gcal
...
...
...
...
...
...
...
Cells: ✅ + one-line evidence • ⚠️ + caveat • ❌ + linked issue • ⏭️ + reason for skip. Never a bare ✅ without evidence.
Provider-specific feature parity audits (e.g. "does the Linear adapter cover all 53 webhook event types"). Those are separate work.
Update this list as scope grows.
Updating this playbook
Append a new failure-pattern row to the table whenever a new class of gap is discovered. Append a new phase section if a phase emerges that isn't covered (e.g. a Phase 8 — Bulk operations if bulk writeback semantics become a real concern). An unrecorded failure mode WILL recur — keep this file current as the live ledger.
