| name | verify-forecasting-docs |
| description | Verify the forecasting docs (quickstart.md, sdk.md, openai.md) by executing every Python code snippet and checking that the surrounding prose accurately describes the actual behavior. Use when docs have changed, when the SDK has changed, or when a new API feature is added. |
Verify Forecasting Docs
Two-phase verification of docs/forecasting/: run every code snippet against the live API, then check that the prose around each snippet matches observed behavior.
Setup
Ask the user for an API key if one hasn't been provided. The project .env may point to staging — bypass it so snippets run against production:
LIGHTNINGROD_BASE_URL="https://api.lightningrod.ai/v1" python ...
Activate the venv before running any Python:
source venv/bin/activate
Phase 1: Execute every code snippet
Read each doc file in turn. For every fenced Python block, run it verbatim (substituting the real API key for "your-api-key"). Capture stdout and any exception.
Cover all files under docs/forecasting/** in the order they are listed in SUMMARY.md.
For each snippet record:
- file and approximate line range
- exit status (pass / error / exception type + message)
- stdout (first 300 chars is enough)
Phase 2: Semantic validation
For each snippet that passed, check the prose immediately before and after it against what you actually observed. Flag any mismatch as a finding.
Common things to check:
| Prose claim | How to verify |
|---|
| "field X is populated when answer_type=Y" | Confirm the field is non-None in the output |
| "answer_type='auto' raises ValueError in predict()" | Run it and confirm the error message matches |
| Parameter descriptions in tables (type, default, description) | Cross-check against client.py and prediction.py source |
| Response field table in sdk.md | Confirm each field is present on the returned PredictionResult |
| Cost/token counts mentioned | Confirm usage fields are present and non-zero |
Source objects have url and title | Confirm with a research call |
answer_type='auto' note says it raises ValueError | Run it and confirm |
Flag findings as:
- ⚠️ wrong — prose makes a claim that contradicts observed behavior
- ⚠️ stale — prose describes something that no longer exists (removed field, renamed param)
- 📝 missing — behavior exists but isn't documented
Report format
## Doc Verification: forecasting docs
**Verdict:** PASS | FAIL
### Snippet results
| # | File | Description | Result |
|---|------|-------------|--------|
| 1 | quickstart.md | OpenAI client minimal | ✅ pass |
| 2 | sdk.md | lr.predict() minimal | ✅ pass |
...
### Semantic findings
- ⚠️ <finding>
- 📝 <finding>
(empty = all prose checked out)
FAIL if any snippet raises an unexpected exception, or if any ⚠️ finding is found. 📝 findings don't fail but should be noted.
