メニュー
Run or monitor the DPLA Python ingest orchestrator. Use when the user says run orchestrator, parallel ingest, ingest status, run hubs, orchestrator dry-run, or retry failed hubs. Covers venv, main entry point, status script, and logs.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
Workflow for generating and sending DPLA monthly pre-ingest emails to hub partners via AWS SES. Use this skill whenever someone asks to: send the monthly ingest email, prepare the DPLA pre-ingest notice, generate the hub email for a given month, preview the pre-ingest email, or do anything involving the pre-ingest email pipeline. Trigger even if the user just says something like "time to send the ingest email" or "it's email day" — context from the project makes the intent clear.
Run Community Webs ingest. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.
Remove DPLA items from the live site and search index in response to takedown requests. Use when the user says take down, delete, remove item(s) from dp.la, or action a takedown request. Accepts DPLA IDs directly or criteria (hub, institution, collection) to generate an ID list. Includes mandatory pre-flight verification, search-before-delete on Elasticsearch, and post-deletion confirmation with count delta verification.
Show key i3.conf config for a hub (provider, harvest.type, harvest.endpoint, schedule, email, setlist). Use when user asks for hub config, harvest type/endpoint, who gets emails, schedule months, or OAI setlist details.
Debug and fix DPLA hub ingestion failures (harvest/mapping/enrichment/jsonl/s3-sync/anomaly). Use when user asks why a hub failed, to debug an ingest failure, check an escalation report, or retry a failed hub/stage.
Show all active and completed ingests in one consolidated view. Use when user asks status of the ingests, ingest status, which hubs are running, show active ingests, or ingest progress.
| name | dpla-orchestrator |
| description | Run or monitor the DPLA Python ingest orchestrator. Use when the user says run orchestrator, parallel ingest, ingest status, run hubs, orchestrator dry-run, or retry failed hubs. Covers venv, main entry point, status script, and logs. |
Run and monitor the Python orchestrator that drives the full ingestion pipeline (harvest → mapping → enrichment → JSONL → anomaly → S3 sync) for one or more hubs, with Slack notifications and parallel execution.
Environment: Always run source .env from repo root before running the orchestrator so JAVA_HOME, SLACK_WEBHOOK, and other vars are set. Ensure the fat JAR is current: from repo root run source .env then sbt assembly before starting the orchestrator (or confirm no Scala changes since last build). Full checklist: AGENTS.md § Environment and build.
source .env
./venv/bin/python -m scheduler.orchestrator.main [options]
Do not use system python3; use ./venv/bin/python so dependencies and environment are correct.
| Goal | Command |
|---|---|
| Current month, all scheduled hubs | ./venv/bin/python -m scheduler.orchestrator.main |
| Specific hubs | ./venv/bin/python -m scheduler.orchestrator.main --hub=wisconsin,p2p |
| Parallel (2–3 hubs at once) | ./venv/bin/python -m scheduler.orchestrator.main --hub=wi,va,mn --parallel=3 |
| Specific month | ./venv/bin/python -m scheduler.orchestrator.main --month=2 |
| Preview only (no run) | ./venv/bin/python -m scheduler.orchestrator.main --dry-run |
| Retry last run's failures | ./venv/bin/python -m scheduler.orchestrator.main --retry-failed |
| Skip harvest (reuse data) | ./venv/bin/python -m scheduler.orchestrator.main --hub=wisconsin --skip-harvest |
| Skip S3 sync | ./venv/bin/python -m scheduler.orchestrator.main --hub=wisconsin --skip-s3-sync |
Per-hub status is written to logs/status/<hub>.status (JSON). Use the status script:
./scripts/status/ingest-status.sh # Table view
./scripts/status/ingest-status.sh --watch # Auto-refresh (e.g. every 30s)
./scripts/status/ingest-status.sh wisconsin p2p # Specific hubs
./scripts/status/ingest-status.sh -v # Verbose (stage history, durations)
./scripts/status/ingest-status.sh --json # JSON (for scripting)
| Resource | Path |
|---|---|
| Orchestrator entry | scheduler/orchestrator/main.py |
| Config | scheduler/orchestrator/config.py; .env for SLACK_WEBHOOK, JAVA_HOME |
| Per-hub status | logs/status/.status |
| Escalation reports | data/escalations/failures-<run_id>.md |
| Email drafts (after run) | logs/hub-emails-<run_id>/ |
Harvests can run 12–24 hours. Use tmux or nohup so the run survives disconnection:
tmux new -s ingest
cd /path/to/ingestion3 && source .env
./venv/bin/python -m scheduler.orchestrator.main --hub=wisconsin,p2p --parallel=2
# Ctrl-B, D to detach; reattach with: tmux attach -t ingest
Slack notifications go to #tech-alerts (and hub-complete to #tech when configured). Failures are written to data/escalations/.