| name | investment-data-project-monitor |
| description | Monitor and safely repair the chenditc/investment_data GitHub Actions pipeline running on the nvdev2 ARC/minikube runner. Use for on-demand nightly health checks, missing daily data, missing qlib releases, ARC health, or investment_data workflow failures. |
Investment Data Project Monitor
Run this on demand, preferably after 21:00 Asia/Shanghai when checking the completed daily pipeline. Verify the immutable data source first, then the GitHub Actions jobs and the downloaded archive/manifest pair. Perform only the explicitly allowed low-risk repairs. Notify via Feishu when the state is abnormal, ambiguous, or needs a risky action.
Current Architecture
- GitHub repository:
chenditc/investment_data, default branch main
- DoltHub repository:
chenditc/investment_data, branch master
- Data table:
final_a_stock_eod_price
- Trading calendar:
ts_trade_day_calendar
- Data workflow:
data_update.yml
- Release workflow:
upload_release.yml, scheduled at 11:01 UTC / 19:01 Asia/Shanghai
- Canonical release pair:
qlib_bin.tar.gz and qlib_bin.manifest.json
- Runner host:
nvdev2
- Minikube profile:
investment-gha
- ARC controller namespace:
arc-systems
- ARC runner namespace:
arc-runners
- Runner scale set:
investment-arc
- Docker graph cache PVC:
investment-data-docker-graph, expected capacity 100Gi
The old serv1 runner, upload-qlib-tar container, host cron, archive path, and proxy are retired. Never use the old repair commands from serv1.
Safety Rules
- Never print or inspect secret values, environment dumps, Kubernetes Secrets, or container environments.
- Never delete or recreate a release, tag, PVC, PV, Docker volume, minikube profile, or ARC installation automatically.
- Never publish a release unless Dolt
max_date equals the expected SSE trade date.
- Never perform a direct release upload. Publication authority belongs only to
upload_release.yml on main.
- Never restart minikube, ARC controller, or listener automatically. Report and notify first.
- Do not rerun a successful workflow merely because a later artifact was intentionally removed.
- Trigger each workflow at most once per monitor run, then wait for and verify its result.
- Prefer observation over repair before 18:30 for data freshness and before 21:00 for release publication.
Check Sequence
- Run the collector:
bash ~/.claude/skills/investment-data-project-monitor/scripts/collect_health.sh
The collector downloads both canonical assets on every run, verifies their GitHub API size and SHA-256 identities, and runs the deployed byte-identical archive validator. A complete healthy, degraded, or not_due report exits 0; an inability to construct a report exits nonzero with empty stdout.
- Read its JSON report. Required healthy state after 21:00 Asia/Shanghai:
dolt.data_fresh is true.platform.healthy is true.- The latest
main data workflow is successful.
- The latest
main upload workflow is successful.
release.valid is true for today's tag.- Exactly one total uploaded asset exists for each canonical name.
release.archive_validation.ok is true; the object is the validator's unmodified JSON document.- No workflow has been in progress for more than 90 minutes.
The reported asset size alone is not evidence of validity. A missing, duplicate, wrong-state, digest-mismatched, stale, malformed, or unsafe archive pair makes the release invalid and, after the due time, degrades overall health.
- If a workflow failed, inspect only the failed run and its failed job logs:
gh run view RUN_ID --repo chenditc/investment_data --json status,conclusion,jobs,url
gh run view RUN_ID --repo chenditc/investment_data --log-failed
Summarize the error. Do not include secrets or long logs in the report or notification.
Allowed Repairs
Stale data after 18:30
Trigger data_update.yml only when Dolt is stale, no update is queued/running, the latest scheduled main update failed/cancelled/is absent, and this monitor run has not already triggered it:
gh workflow run data_update.yml --repo chenditc/investment_data --ref main
Wait for completion, then rerun the collector.
Missing or invalid release after 21:00
Trigger upload_release.yml only when Dolt is fresh, no upload is queued/running, today's canonical pair is invalid, the scheduled main upload failed/cancelled/did not run, and this monitor run has not already triggered it:
gh workflow run upload_release.yml --repo chenditc/investment_data --ref main -f operation=publish
Wait for completion, then rerun the collector. If the workflow succeeds but the downloaded pair remains invalid, notify instead of attempting direct upload.
Feishu Notification
Send a notification for any degraded state after its due time, any repair attempt, repeated failure, stuck workflow, low disk space, or action requiring human judgment:
~/.claude/skills/investment-data-project-monitor/scripts/notify_feishu.sh "investment_data nightly monitor
当前时间: ...
状态: ...
expected trade date: ...
Dolt max_date: ...
data_update: ...
upload_release: ...
release archive validation: ...
ARC/minikube: ...
已执行的安全修复: ...
建议人工动作: ..."
The helper, its recipient, arguments, secret lookup, retry timing, and message behavior are unchanged. Do not send a normal-success notification. Never include secrets, environment dumps, or long logs.
Fail-safe repository rollback
A full repository revert is ordered and fail-closed:
- Run
gh workflow disable upload_release.yml --repo chenditc/investment_data.
- Query
gh workflow view upload_release.yml --repo chenditc/investment_data --json state --jq .state and require the exact result disabled_manually.
- Query both
upload_release.yml and data_update.yml runs and wait until every queued or in-progress job using the shared Dolt volume has drained.
- Perform the full repository revert.
- Run
gh workflow view upload_release.yml --repo chenditc/investment_data --json state --jq .state again and require disabled_manually.
- Do not re-enable publication until validator-backed digest pinning, workflow authority, and shared concurrency/filesystem locking are restored and verified end to end.
The revert may move the convenience latest image and therefore affect data update, but it cannot publish while the upload workflow is disabled. Draining is mandatory because a full revert may remove the shared lock and concurrency group. Already accepted release assets are untouched. An interrupted historical repair may complete only through the fixed repair-2026-07-20 operation, and the stale backup is never auto-restored. The deployed monitor is separate external state; roll it back only with the tracked ops/investment-data-project-monitor/deploy.sh rollback, never as part of the repository revert.
Final Report
Report the Asia/Shanghai time and overall state; expected and actual Dolt dates; workflow states and URLs; canonical asset identities and release.archive_validation; ARC/minikube/PVC/disk health; stuck runs; repairs and post-repair verification; Feishu outcome; and remaining human action.
Principle: validate the actual release bytes against immutable provenance before declaring publication healthy.
