ar-io-gateway-operator

name

ar-io-gateway-operator

description

Operate any AR.IO node deployment — architecture, daily ops, diagnostics, and recurring pitfalls that apply to every operator. Use whenever the user asks about gateway health, indexing lag, ClickHouse pipeline, GraphQL routing, ArNS resolution, data retrieval and caching, bundle unbundling, webhook fan-out, container restarts, image rebuilds, disk pressure, HTTPSig overhead, or anything operationally generic to running an AR.IO node — even when they don't name a specific service ("is it healthy?", "why is it slow?", "did the build pick up?", "how full is the disk?"). For deployment-specific knowledge (domains, repo paths, sidecar config, this-machine numbers), pair this with the deployment-overlay skill (e.g. `vilenarios-gateway`) if one exists.

AR.IO Gateway Operator (generic)

You are operating an AR.IO node — an Arweave gateway that indexes, caches, serves, and signs blockchain data. Deployments share the same architecture but differ in domains, paths, and which sidecars are present. Run scripts/health-check first every session: it prints a one-screen snapshot you can compare against the prior state before deciding to act.

If a deployment-specific overlay skill is loaded (e.g. vilenarios-gateway), it provides the local repo paths, domains, and which sidecars exist. Treat the overlay as authoritative for those facts.

Run first: scripts/health-check

# from the ar-io-node repo root
.claude/skills/ar-io-gateway-operator/scripts/health-check

What it checks and why:

Section	What "good" looks like	If wrong, jump to
containers	All core ar-io-node containers Up	"Bring everything up"
disk	/ < 80%, /data (if separate) has headroom	"Disk pressure"
gateway healthcheck	{"status":"ok",...}	"When something breaks"
indexer height vs chain head	lag ≤ 5 blocks	"Indexing falling behind"
ClickHouse rows	If streaming enabled: new_blocks > 0 and growing	"Streaming tables empty"
core errors last 5 min	< ~10 (operational noise: peer fetches, aborts)	grep deeper
image SHAs	If on a dev branch, must differ from :latest digest	"Building from source"

The script only covers the core ar-io-node containers defined in this repo's docker-compose. Sidecar checks belong in a deployment-overlay skill that calls this script then adds its own checks on top.

How the gateway works (the model an operator needs)

Knowing the data flows tells you where to look when symptoms appear. Source of truth is src/system.ts (DI wiring) and docs/; this section captures the operationally-load-bearing facts.

Request flow

client → (optional edge: nginx/CDN) → envoy :3000 → core :4000 → response with X-AR-IO-* trust headers
                                                              ↘ webhook fan-out → optional sidecars

Every served data response carries trust headers (X-AR-IO-Verified, X-AR-IO-Trusted, X-AR-IO-Stable, X-AR-IO-Hops, plus X-Arweave-* tag passthrough). Those headers are what HTTPSig signs — see src/middleware/httpsig.ts TRIGGER_HEADERS. Endpoints with no trust trigger (/ar-io/info, /ar-io/healthcheck, /graphql errors) are deliberately not signed.

Data retrieval — composite source with fallback chain

Configurable via .env. Defaults:

• ON_DEMAND_RETRIEVAL_ORDER=trusted-gateways,ar-io-network,chunks-offset-aware,tx-data — what /raw/<id> and /<id> walk through on a cache miss
• BACKGROUND_RETRIEVAL_ORDER=chunks — used by the unbundler when fetching bundle bodies

Each source's failures are logged with class so you can grep (GatewaysDataSource, TxChunksDataSource, ChunksDataSource, etc.). Persistent failures from one tier are normal — the chain is designed to fall through. Look for Failed to fetch chunk data from any source / all sources failed as the real failure signal, not individual tier warnings.

Three response headers worth knowing: X-Cache: HIT|MISS (gateway-side contiguous cache), X-AR-IO-Hops (how many AR.IO peers were traversed before this gateway served), X-AR-IO-Trusted / X-AR-IO-Verified (whether the data has been hash-verified against the chain).

Indexing pipeline

Two write paths run side-by-side:

1. SQLite primary indexer (always on) — BlockImporter worker imports blocks one at a time into new_blocks/new_transactions (the unstable head, ~last 18 blocks), and a separate process promotes them to stable_blocks/stable_transactions once they're past the reorg horizon. ANS-104 bundle unbundling is gated by ANS104_UNBUNDLE_FILTER (often narrow). Database access is in a worker thread (StandaloneSqlite) — never call SQLite synchronously from the main thread.

2. ClickHouse pipeline (optional, --profile clickhouse) — Stable rows are exported to parquet then imported by clickhouse-auto-import (hourly cycles) into staging_* and finally transactions. Streaming pipeline (PR #699 / clickhouse-streaming-unstable-head branch) mirrors the unstable-head SQLite writes directly into ClickHouse new_blocks / new_transactions so GraphQL can query the head without waiting for the parquet round-trip. Activated by CLICKHOUSE_STREAMING_ENABLED=true. Reorgs trigger bounded ALTER TABLE ... DELETE WHERE on the new_* tables.

GraphQL routing

/graphql reads from up to three legs that get merged: stable transactions table in CH, streaming new_transactions in CH, and the SQLite new_* fallback. With CLICKHOUSE_GQL_SKIP_SQLITE_READS=true, the SQLite leg is dropped to a tight-timeout fallback only. With GRAPHQL_ON_DEMAND_RESOLUTION_ENABLED=true (default), transaction(id) queries that miss the local DB will fall back to extracting from ANS-104 bundle binaries on demand — single-ID only, capped by GRAPHQL_ON_DEMAND_RESOLUTION_MAX_CONCURRENT.

ANS-104 unbundling pipeline

Whenever a bundle TX matches ANS104_UNBUNDLE_FILTER, it walks a four-queue chain. Knowing the chain is essential for diagnosing throughput regressions — a single queue at cap can mask everything downstream of it.

bundleDataImporter ─→ ans104Unbundler ─→ ans104DataIndexer ─→ dataItemIndexer ─→ SQLite writer
   download workers      parser workers     post-parse buffer     pre-write buffer    single-writer
   (ANS104_DOWNLOAD_…)   (ANS104_UNBUNDLE_)  (ANS104_DATA_…_SIZE)  (DATA_ITEM_…_SIZE)

Tune knobs (all env-overridable):

Knob	Default	What it controls
ANS104_DOWNLOAD_WORKERS	5	Parallel bundle downloads. Network-bound; overprovisioning is cheap
ANS104_UNBUNDLE_WORKERS	1	Parallel parser worker_threads. CPU + network bound
DATA_ITEM_INDEXER_WORKER_COUNT	1	Fastq concurrency for indexDataItem; SQLite is single-writer so gains are bounded
MAX_DATA_ITEM_QUEUE_SIZE	100000	Soft gate: shouldUnbundle() returns false above this on either indexer queue
DATA_ITEM_INDEXER_QUEUE_SIZE	500000	Hard cap on dataItemIndexer; items past this get dropped silently
ANS104_DATA_INDEXER_QUEUE_SIZE	500000	Hard cap on ans104DataIndexer

The buffer between soft gate and hard cap is where data loss happens. With defaults the gate engages at 100 K but the hard cap is 500 K — a 5× buffer. Under heavy load, in-flight parser workers can flood items into the queue after the gate engages but before the indexer drains, exceeding the hard cap and silently dropping items. Symptom: data_items_dropped_total{queue_name="dataItemIndexer"} climbs while bundles_unbundled_total keeps increasing — bundles look "indexed" but their items aren't searchable.

Mitigations:

• Tighten MAX_DATA_ITEM_QUEUE_SIZE so the gate engages well before the hard cap (e.g., 25 000 with 500 000 hard caps, giving a 20× buffer).
• Raise the hard cap if memory permits (each item slot ≈ 1–2 KB).
• Don't add parser workers when data_items_dropped_total is non-zero — more workers = bigger floods past the gate.

Pipeline diagnostic: a single curl -sf .../ar-io/__gateway_metrics | grep -E 'queue_length|data_items_dropped|bundles_unbundle_in_flight' tells you which queue is the bottleneck.

Caching layers (in order, fastest first)

Layer	Where	TTL	Notes
In-process kv	core memory	varies per resolver	ArNS names cache, etc.
Redis	redis:7 container	varies	Some ArNS / network state
Contiguous data store	${CONTIGUOUS_DATA_PATH}/	unbounded	Files keyed by content hash; cleanup workers prune
Response cache hints	Cache-Control headers in response	per-route	e.g., /raw/:id HIT serves with max-age=43200
Edge cache (optional)	nginx/Varnish/CDN in front	per-zone	Only present if the operator deployed one — gateway doesn't run one itself

X-Cache: HIT on a /raw/<id> response means the gateway's contiguous data store served bytes without going to peers/Arweave. The edge layer is separate — nginx's X-Cache-Status: HIT is a different layer.

ArNS resolution

CompositeArNSResolver walks resolvers in order: TrustedGatewayArNSResolver (asks TRUSTED_ARNS_GATEWAY_URL, default https://__NAME__.turbo-gateway.com), then OnDemandArNSResolver (queries the on-chain ario-arns and ario-ant programs directly via SOLANA_RPC_URL). The base name set is paginated into an in-memory ArNSNamesCache at boot and refreshed on a debounce. Resolved IDs may be Arweave TXs (route to data path) or, on the streaming-head branch, IPFS CIDs (route to /ipfs/<cid> via the Kubo sidecar). Unknown names log Unable to resolve name against all resolvers — that's normal user-error traffic, not a service failure.

/<name> and /<name>/<path> requests carry X-ArNS-* trust headers in the response. Manifest path resolution still uses StreamingManifestPathResolver; the "from index" path is not implemented yet (logs warn not implemented then falls back to data-side resolution, which works).

Network identity, observer, and incentives

A gateway is a registered participant in the AR.IO network, not just a piece of software. Two distinct Solana identities matter:

• Operator — AR_IO_WALLET (pubkey, base58) plus a signing key as either SOLANA_KEYPAIR_PATH (JSON file) or SOLANA_PRIVATE_KEY (base58 secret, Phantom-export form). Stakes ARIO tokens (20,000 minimum = 20000000000 mARIO since v3.0.0), receives rewards, signs cranker instructions when ENABLE_EPOCH_CRANKING=true.
• Observer — OBSERVER_WALLET (pubkey) plus a signing key as either OBSERVER_KEYPAIR_PATH or OBSERVER_PRIVATE_KEY. Submits save_observations instructions and signs HTTPSig responses. The two roles are deliberately separable so the observer key can sit on the gateway box while the operator/staking key stays in cold storage. Setting both forms (path and PK string) for the same role is rejected at startup.

The Solana key designated for the observer role is the HTTPSig signing key. The observer's address is already registered in the on-chain Gateway Registry (GAR), so verifiers can confirm a signed response traces back to a staked, registered gateway with a single GAR lookup. No attestation document is uploaded to Arweave on the Solana path — the chain is the source of truth.

Four Solana program IDs configure which network the gateway talks to: ARIO_CORE_PROGRAM_ID, ARIO_GAR_PROGRAM_ID, ARIO_ARNS_PROGRAM_ID, ARIO_ANT_PROGRAM_ID. Plus SOLANA_RPC_URL (premium provider strongly recommended in production; public mainnet-beta throttles hard). GET /ar-io/info echoes the resolved programIds object, which is the fastest way to confirm a running gateway is pointed at the network you think it is.

Observer service knobs:

• RUN_OBSERVER=true (default) — runs the observer alongside the gateway
• SUBMIT_CONTRACT_INTERACTIONS=true (default) — submits generated reports on-chain
• OBSERVATIONS_PER_GATEWAY (default 3) — how many times each gateway is observed per epoch (majority vote)
• OBSERVATION_WINDOW_FRACTION (default 0.5) — what fraction of the daily epoch window observations are spread across
• REFERENCE_GATEWAY_MIN_CONSECUTIVE_PASSES / REFERENCE_GATEWAY_MIN_EPOCH_COUNT — eligibility thresholds for using a network gateway as a reference

Epoch cadence is daily. Observation reports drive reward eligibility. Operators may themselves be selected as observers for other gateways.

Network onboarding (operator side, only needed once):

• npm install -g @ar.io/sdk — provides the ar.io CLI binary. Use ar.io get-gateway --address <pubkey> to read state, ar.io join-network ... to register.
• The CLI accepts the operator key as either --wallet-file <path> (JSON) or --private-key '<base58>' (Phantom export — bs58.decode runs in-process).
• Every Solana command needs -t solana and the four program IDs (--core-program-id / --gar-program-id / --arns-program-id / --ant-program-id) plus --rpc-url. As of @ar.io/sdk@^4.0.0-solana.22 the bundled "mainnet" constants are still placeholders, so passing --mainnet alone resolves to non-existent addresses — always pass the explicit program IDs the network team publishes for whichever environment you're targeting (mainnet, staging-devnet, local devnet).
• Register via the CLI or the gateways.ar.io portal: operator wallet, observer wallet, FQDN, Properties Tx ID, label, stake (--operator-stake 20000 in whole ARIO — the CLI converts to mARIO internally via requiredMARIOFromOptions → new ARIOToken(...).toMARIO()).
• Gateway must be reachable at the registered FQDN with valid TLS, AND configured for the same programIds set you joined with. The usual cause of a "I joined but nothing's working" report is a gateway whose .env still pins the old network's program IDs.
• Optional delegated staking — operators can accept delegations with a configurable reward share. --min-delegated-stake is mARIO directly (no ARIO→mARIO conversion); --operator-stake is the only stake-input flag that takes whole ARIO.

Serving content at the root path

Two mutually-relevant env vars decide what / serves:

• APEX_TX_ID — pin a specific Arweave TX as the apex. Cache-Control bounded by CACHE_APEX_MAX_AGE with must-revalidate so apex rotations propagate.
• APEX_ARNS_NAME — resolve an ArNS name as the apex. Comma-separated values position-map to ARNS_ROOT_HOST entries when there's more than one host.
• ARNS_ROOT_HOST — the gateway's primary domain(s) for ArNS subdomain resolution.

Filters and webhooks (the gateway's contract with sidecars)

Three filters compose JSON expressions (tags, attributes, or/and/always/never) — see docs/filters.md:

• ANS104_UNBUNDLE_FILTER — which bundles get unbundled
• ANS104_INDEX_FILTER — which data items inside matched bundles get persisted
• WEBHOOK_INDEX_FILTER — which indexed items trigger webhook emission

Webhook events fired: data-cached, tx-indexed, ans104-data-item-indexed. WEBHOOK_TARGET_SERVERS is a comma-separated list of URLs that receive POSTs. If you change a filter to broaden indexing, expect proportionally higher webhook volume.

Common operations

Bring up

# from the ar-io-node repo root
docker compose --profile clickhouse up -d

The --profile clickhouse flag matters: without it, ClickHouse and the auto-import container stay down and the streaming pipeline goes silent. Other profiles (grafana, bundler, litestream, ao, hb) gate their own services.

Wait for ready:

until curl -sf http://localhost:4000/ar-io/healthcheck >/dev/null; do sleep 2; done

Rebuild from local source (after dev-branch code changes)

docker compose --profile clickhouse build core clickhouse-auto-import
docker compose --profile clickhouse up -d --force-recreate core clickhouse-auto-import

--force-recreate is required — without it, compose sees the same image name (:latest) and skips recreation even though the image digest changed. Confirm with docker inspect <container> --format '{{.Image}}' — the SHA must differ from before.

Disk pressure recovery (in order of safety)

docker system df                        # see what's hogging
docker builder prune -af                # build cache (almost always biggest)
docker container prune -f               # stopped containers
docker image prune -af                  # untagged images
sudo journalctl --vacuum-size=500M      # if /var/log/journal is large

Avoid docker volume prune -af without listing first (docker volume ls -f dangling=true) — there can be old SQLite/CH data in dangling volumes worth keeping.

Block / unblock content (gateway admin API)

ADMIN=$(grep '^ADMIN_API_KEY=' .env | cut -d= -f2-)
TX=<arweave-tx-or-ipfs-cid>

# Block (the gateway has PUT only — there is no DELETE handler today)
curl -sf -X PUT "http://localhost:4000/ar-io/admin/block-data" \
  -H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' \
  -d "{\"id\":\"$TX\",\"hash\":\"$TX\",\"source\":\"manual\",\"notes\":\"...\"}"

If a moderation sidecar is present (and wired via WEBHOOK_TARGET_SERVERS), prefer routing the block through it — depending on the sidecar's config, it may also bust an edge cache after blocking.

HTTPSig overhead (for self-measurement)

Read the in-process histogram for the truest per-request signing cost — no benchmarking needed:

curl -sf http://localhost:4000/ar-io/__gateway_metrics | grep '^httpsig_signing_duration'
# mean per sign = sum / count  (typically 70-120 µs; under 500 µs at p100)

Don't draw conclusions from /ar-io/info or /ar-io/healthcheck benchmarks — those carry no trust-trigger headers and are never signed.

Pitfalls (apply to any deployment)

1. clickhouse-auto-import logs Failed to connect to core port 4000 after 0 ms — the script's connection state goes stale when core is recreated mid-cycle, and auto-import sleeps 1 h between cycles, so the bad state lingers. Recreating just the auto-import container clears it; manual curl http://core:4000/... from inside the container will succeed even while the script is failing.
2. Streaming tables empty (new_blocks / new_transactions = 0) despite CLICKHOUSE_STREAMING_ENABLED=true — running the published :latest core image instead of a local build. The streaming columns/tables are populated only by the dev-branch code. Rebuild + force-recreate.
3. Webhook error flood (getaddrinfo ENOTFOUND <sidecar-host>) — a configured sidecar isn't running. Bring it up. Don't strip the target unless intentionally retiring the sidecar — those targets are how moderation and provenance subsystems work.
4. Bundle unbundling looks idle (low matched count) — ANS104_UNBUNDLE_FILTER is intentionally narrow on most operator configs (e.g., one App-Name + one owner). Confirm via /ar-io/info. Not a bug.
5. Gateway has no DELETE /ar-io/admin/block-data handler — only PUT. Sidecars that send DELETE (e.g. an unblock action) will see failures. Document but don't try to "fix" by changing the sidecar — the gateway is the side that needs the route.
6. /ar-io/info and /ar-io/healthcheck are unsigned — they carry no trust-trigger headers, so HTTPSig signing skips them. Use /raw/<id> against a cached tx for HTTPSig benchmarks.
7. docker logs <core> | tail can dump megabytes of binary — the streaming pipeline sometimes logs cert chains as raw byte arrays. Filter with grep -viE '"buffer":|raw bytes|^[\[0-9, ]+$'.
8. Five-edit dance for adding a database method — SQL statement, worker impl in StandaloneSqlite, queue wrapper in main DB class, case in worker message handler, interface in types.d.ts. Skip any one and you get silent failures.
9. Observer restart-loops with Epoch 0 PDA not found at <pda> — has prescribe_epoch run yet? — the network you're pointed at hasn't had its first epoch initialized. The observer can't bootstrap without entropy from epoch[N].prescribed_observers, and that PDA only exists after a cranker has called create_epoch. Confirm with ar.io get-current-epoch ... — "Epoch 0 not found" means network-side, not gateway-side. Wait for an active cranker (yours or another operator's) to bootstrap epoch 0.
10. Cranker started cleanly but never logs any [crank:*] activity — EpochSettings.enabled === false on the target network. The cranker bails silently at epoch-cranker.ts:173 (debug log only) so it doesn't burn SOL submitting ix against a paused network. Verify with ar.io get-epoch-settings .... No fix on the gateway side; this is a network-operations state.
11. ArNS names return 404 even though ar.io get-arns-record --name <name> returns the record fine — SDK version drift. ArNSNamesCache.hydrate paginates the on-chain registry; an SDK pin that's significantly older than the deployed ario-arns program may parse paginated responses incorrectly, hydrating "successfully" but with most entries missing. Logs show Successfully hydrated ArNS names cache quickly (~few seconds for thousands of records) followed by Base name not found in ArNS names cache on lookups. Fix: bump @ar.io/sdk in package.json to the latest ^4.0.0-solana.* and rebuild.
12. /ar-io/info shows the OLD network's programIds after a migration / image swap — the .env has explicit CORE_IMAGE_TAG / ENVOY_IMAGE_TAG / OBSERVER_IMAGE_TAG pins that shadow the compose defaults. Updating only the compose defaults won't move a gateway whose .env still pins old image tags. Either update both, or remove the explicit pins from .env and let compose defaults win.

When something breaks: where to look

Symptom	First check
5xx from gateway	docker logs --tail 50 <core-container> (filtered), then envoy logs
Indexer falling behind	SQLite MAX(height) per block_importer log, peer health, /data free space
ClickHouse stale	docker compose --profile clickhouse logs --tail 50 clickhouse-auto-import; verify curl-to-core works from inside the container
Streaming table stuck at 0	image SHA — almost always the published :latest rather than local build
Admin API 401/404	ADMIN_API_KEY mismatch with .env
Webhook silent	WEBHOOK_TARGET_SERVERS in gateway .env; sidecars must be on the same docker network and reachable by service name
/data filling	du -h -d 1 /data | sort -hr — usually the contiguous cache
Observer not submitting save_observations	docker logs ar-io-node-observer-1 --since 10m | grep -iE 'crank|epoch'. Combine with ar.io get-current-epoch ... to distinguish gateway-side issues from a paused/uninitialized network
ArNS resolution returning 404 for names that demonstrably exist	Check the SDK pin in the container: docker exec ar-io-node-core-1 /nodejs/bin/node -e "console.log(require('@ar.io/sdk/package.json').version)". Drift behind the deployed program causes silent partial cache hydration
/ar-io/info shows wrong programIds after migration	.env likely has explicit *_IMAGE_TAG lines pinning the old images. Update them or remove to let compose defaults take over

Metrics worth watching

Quick-scan list of the prometheus metrics that surface most operator-grade issues. All exposed at /ar-io/__gateway_metrics on core's HTTP port.

Pipeline throughput + saturation

Metric	Surfaces
bundles_unbundled_total (rate)	bundle-level throughput
data_items_indexed_total{parent_type="transaction"} (rate)	data-item write rate (SQLite writer capacity)
bundles_unbundle_in_flight	parser workers currently active — if pinned at worker count for minutes, suspect cascade hangs
queue_length{queue_name="…"}	the 4-queue chain — see "ANS-104 unbundling pipeline"
data_items_dropped_total{queue_name="…"}	silent data loss at indexer hard cap. Any nonzero rate is a config issue, not a transient
bundles_unbundle_skipped_total{reason="…"}	high_queue_depth (soft gate engaged), queue_full (BRW pushing past cap), no_workers (config error)

Per-source health

Metric	Surfaces
get_data_stream_successes_total{class,source}	which cascade tier / which peer actually serves bytes
get_data_stream_errors_total{class,source}	tier/peer failure rates. Persistent 100 % errors on a peer mean it's dead but the network process still advertises it
get_data_errors_total{…}	pre-stream errors (connection refused, 4xx, 5xx)

Sign of saturation, not failure

Metric	What it means
nodejs_event_loop_utilization	0–1, sampled at scrape. Near 1.0 = main thread pegged
nodejs_eventloop_lag_p99_seconds	tail latency on the event loop. > 0.1 s under load = something blocking on the main thread
process_resident_memory_bytes	overall memory growth; sudden steps usually map to a queue filling
httpsig_signing_duration_seconds_*	per-response signing cost (the histogram is more honest than benchmarking)

Performance-interpretation guidance

• Sample windows lie. A 60–90 s rate can show 0/hr or 60K/hr depending on whether you caught the soft-gate cycle in its open or closed half. Trust averages over many minutes, not single snapshots.
• Saturated workers ≠ broken workers. in_flight = N for the configured worker count with throughput still climbing is the healthy steady state; only worry if the corresponding _total counter is also flat for many minutes.
• The cascade goes through phases. When trusted-gateway A goes hot-cache for an hour, throughput spikes; when it rotates out, throughput dips. Aggregate over a day, not a window.

Adjacent skills (use these, don't reinvent)

• release — handles version bumps, image SHA pinning, tags, GitHub releases.
• testing — picks the right test layer (unit / property / e2e / auto-verify / parquet integration / load test) for a given change.
• Deployment-overlay skill (e.g. vilenarios-gateway) — adds the local-machine specifics (domains, paths, sidecar config) on top of this generic skill.

This skill covers the operations layer: what's running, where, and how to keep it that way. For code-level guidance see CLAUDE.md in the relevant repo.