rsky-pds-crawler-notify-hang

name	rsky-pds-crawler-notify-hang
description	Fix rsky-pds createRecord/putRecord hanging indefinitely with no error logs. Use when: (1) createSession succeeds but createRecord never returns, (2) No error appears in PDS logs despite request hanging for 60+ seconds, (3) PDS health endpoint works but write operations hang, (4) readiness probes intermittently fail on rsky-pds pod. Root cause: PDS_CRAWLERS env var configures crawler notification URLs that are called via reqwest HTTP client WITHOUT any timeout in crawlers.rs. If the crawler URL is unreachable from the cluster, every write hangs forever. Also: setting PDS_CRAWLERS="" produces [""] (one empty string element) which causes a "builder error" instead — must fully unset the env var.
author	Claude Code
version	1.0.0
date	"2026-03-21T00:00:00.000Z"

rsky-pds Crawler Notification Hang

Problem

All write operations on rsky-pds (createRecord, putRecord, deleteRecord) hang indefinitely with no error output. The PDS appears healthy (health endpoint returns 200, createSession works) but any operation that modifies the repo never completes.

Context / Trigger Conditions

• createSession returns a valid JWT instantly
• createRecord or putRecord with that JWT hangs forever (60s+ timeout)
• PDS logs show only Rocket has launched from http://0.0.0.0:8000 with no error
• _health endpoint returns {"version":"0.3.0-beta.3"} (or similar)
• Kubernetes readiness probes may intermittently fail due to the pod being overwhelmed by hung requests
• The PDS_CRAWLERS env var is set (e.g., https://bsky.network)

Solution

Quick Fix (Runtime)

Unset the PDS_CRAWLERS environment variable entirely:

# IMPORTANT: Use the trailing dash to UNSET, don't set to empty string
kubectl set env deployment/rsky-pds -n <namespace> PDS_CRAWLERS-

# Wait for rollout
kubectl rollout status deployment/rsky-pds -n <namespace> --timeout=120s

Do NOT set PDS_CRAWLERS="" — the env_list() parser splits on commas, so an empty string produces vec![""] (one empty-string element), which causes a "builder error" when reqwest tries to build a URL from "".

Code Fix (Permanent)

In rsky-pds/src/crawlers.rs, add a timeout to the reqwest client:

let client = reqwest::Client::builder()
    .user_agent(APP_USER_AGENT)
    .timeout(std::time::Duration::from_secs(5))  // Add this
    .connect_timeout(std::time::Duration::from_secs(3))  // Add this
    .build()?;

Also consider making the error non-fatal (use let _ = to ignore failures instead of propagating with ?), since crawler notification is not critical to the write operation succeeding.

Root Cause Analysis

The write path in rsky-pds is:

1. createRecord -> process_writes -> format_commit (DB, works fine)
2. sequence_commit -> sequence_evt -> crawlers.notify_of_update() (HANGS)

notify_of_update() in crawlers.rs creates a reqwest::Client with no timeout configured and POSTs to each crawler URL. From within a GKE cluster, the request to https://bsky.network/xrpc/com.atproto.sync.requestCrawl may hang at the TCP level with no response, causing the entire write to block.

Key files:

• rsky-pds/src/crawlers.rs:30-61 — notify_of_update() method
• rsky-pds/src/sequencer/mod.rs:215 — calls crawlers.notify_of_update()
• rsky-pds/src/apis/com/atproto/repo/create_record.rs:100 — acquires sequencer lock
• rsky-common/src/env.rs:28-33 — env_list() parser that splits empty strings

Verification

After unsetting PDS_CRAWLERS:

# Should return in <10 seconds instead of hanging
curl -X POST "$PDS_URL/xrpc/com.atproto.repo.createRecord" \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/json" \
  -d '{"repo":"$DID","collection":"app.bsky.feed.post","record":{"$type":"app.bsky.feed.post","text":"test","createdAt":"2026-01-01T00:00:00Z"}}'

Notes

• This affects ALL rsky-pds deployments that set PDS_CRAWLERS to an unreachable URL
• The upstream TypeScript atproto PDS has proper timeouts; this is an rsky-specific bug
• The notify_of_update() has a 20-minute throttle, so the first write after startup always triggers the notification (and hangs if unreachable)
• The sequencer write lock at line 99 means a single hung notification blocks ALL subsequent write requests, not just the one that triggered it