| name | trace-propagation |
| description | Trace a data value through kong's pipeline hop-by-hop to find where propagation stalls |
Activation Criteria
Use this skill when:
- A value (APR, TVL, snapshot field) changed at its source but isn't showing on the public API
- After deploying an ingest/webhook fix, to verify data propagates end to end
- The user asks to "track propagation", "find the bottleneck", or "check where the data is stuck"
Requirements
Environment (in ~/.env, exported into login shells):
KONG_PROD_RO_REPLICA_DB — read-only replica of the live database. It is NOT a
separate dev environment; what you read here is what prod has (modulo replication lag).KONG_REDIS_RO_URL — read-only connection to the prod REST cache.gh CLI authenticated for yearn/kong (workflow run history).
Sourcing gotcha: values containing & (postgres URLs) must be quoted in ~/.env or
set -a; source ~/.env silently fails to set them (the & backgrounds the assignment).
Values are quoted now; keep them quoted.
The propagation chain
Values flow through these hops. Each hop has a probe (see probes.sh in this skill's
directory) and a freshness stamp. The method is binary search: find the last hop with
the correct value and the first hop without it — the bottleneck is between them.
| # | Hop | Probe | Advances when |
|---|
| 1 | Source service (per config/subscriptions.yaml) | probes.sh source <url> | live — computes at call time |
| 2 | output table | probes.sh output <chainId> <address> <label> | ingest fanout runs the webhook |
| 3 | snapshot table (hook->'performance') | probes.sh snapshot <chainId> <address> | fanout runs the snapshot hook |
| 4 | Redis REST cache (rest:snapshot:{chainId}:{address}) | probes.sh redis <chainId> <address> | refresh-cache.yml workflow runs |
| 5 | Public REST API / CDN edge | probes.sh rest <chainId> <address> | CDN TTL expires (s-maxage=900) |
Start by resolving the label to its subscription in config/subscriptions.yaml
(source URL, abiPath, filter addresses). Then probe hops in order and build a table:
one row per hop, showing the value and its timestamp. Name the stale hop and its cause.
Fallback when DB credentials are unavailable: the public GraphQL API
(https://kong.yearn.fi/api/gql) reads the live DB — probes.sh gql <chainId> <address>
for snapshots, and the timeseries query for output series (pass timestamp: — results
come back ascending, so without it you get the oldest points).
Interpreting what you see
- Compare like with like: output-vs-output, snapshot-vs-cache. An
output row and a
snapshot estimate stamped near the same time can legitimately differ — the snapshot
hook reads whatever webhook batch is loaded at hook time, and intermediate batches get
replaced (below). That skew is normal, not corruption.
- Day-bucket upserts:
output keeps one row per (address, label, component, day).
A new batch replaces the same day's rows — earlier intraday batches "disappear".
Don't treat a vanished batch as data loss.
- Deterministic vs live components: components derived from chain state at the
trigger block (e.g.
debtRatio) match exactly across batches at the same block;
live-computed ones (APRs from external APIs) drift between calls. Use this to tell
batch-timing skew from real corruption. Digit-for-digit equality is the standard for
"same batch".
- Cadences: prod fanout runs every ~15–30 min (don't infer cadence from historical
output rows — day-bucket upserts keep only each day's last batch, which makes it look
daily; read snapshot.block_time advancing instead). The config/abis.yaml cron
start: flag only controls BullMQ job registration at ingest boot — prod is driven
elsewhere. refresh-cache.yml is scheduled */30 * * * * but GitHub Actions delivers
it every 1–3 h in practice — check gh run list --workflow refresh-cache.yml before
concluding anything is broken. The CDN adds up to 15 min on top of Redis.
- Neon sleeps: the replica suspends when idle; the first connection fails with
"Control plane request failed".
probes.sh retries automatically.
Watch mode
To wait for the public API to catch up, run probes.sh watch <chainId> <address> <value-prefix>
as a background task — it polls every 60s and exits when the served value starts with the
prefix (or after 30 min).
Guardrails
- Every probe is read-only. Keep it that way.
- The one known unstick action —
gh workflow run refresh-cache.yml --repo yearn/kong —
is a mutation. Present it as an option; never trigger it without the user's explicit
go in this conversation.
- Never print credential values (connection strings contain passwords). Refer to env vars
by name;
probes.sh handles redaction.
