القائمة
Run Community Webs ingest from SQLite DB. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
Workflow for generating and sending DPLA monthly pre-ingest emails to hub partners via AWS SES. Use this skill whenever someone asks to: send the monthly ingest email, prepare the DPLA pre-ingest notice, generate the hub email for a given month, preview the pre-ingest email, or do anything involving the pre-ingest email pipeline. Trigger even if the user just says something like "time to send the ingest email" or "it's email day" — context from the project makes the intent clear.
Run Community Webs ingest. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.
Remove DPLA items from the live site and search index in response to takedown requests. Use when the user says take down, delete, remove item(s) from dp.la, or action a takedown request. Accepts DPLA IDs directly or criteria (hub, institution, collection) to generate an ID list. Includes mandatory pre-flight verification, search-before-delete on Elasticsearch, and post-deletion confirmation with count delta verification.
Show key i3.conf config for a hub (provider, harvest.type, harvest.endpoint, schedule, email, setlist). Use when user asks for hub config, harvest type/endpoint, who gets emails, schedule months, or OAI setlist details.
Debug and fix DPLA hub ingestion failures (harvest/mapping/enrichment/jsonl/s3-sync/anomaly). Use when user asks why a hub failed, to debug an ingest failure, check an escalation report, or retry a failed hub/stage.
Show all active and completed ingests in one consolidated view. Use when user asks status of the ingests, ingest status, which hubs are running, show active ingests, or ingest progress.
| name | dpla-community-webs-ingest |
| description | Run Community Webs ingest from SQLite DB. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB. |
Run the Community Webs ingest workflow: export SQLite DB → JSONL → ZIP → harvest → (optional) full pipeline. Internet Archive sends a SQLite database; the scripts handle the intermediate export step automatically.
Environment: Source .env before running so JAVA_HOME, DPLA_DATA, I3_CONF are set. Scripts that source common.sh load $I3_HOME/.env when present. Full checklist: AGENTS.md § Environment and build.
$DPLA_DATA/community-webs/originalRecords/ (or provide path via --db=)../scripts/harvest/community-webs-ingest.sh./scripts/harvest/community-webs-ingest.sh --full./scripts/harvest/community-webs-ingest.sh --full --update-conf_SUCCESS in harvest/mapping/enrichment/jsonl dirs.| Script | Purpose |
|---|---|
community-webs-export.sh | Export DB → JSONL → ZIP; validates schema before zipping |
community-webs-ingest.sh | Orchestrates export + harvest (+ optional full pipeline) |
community-webs-export.sh:
--db=PATH — Explicit DB path (default: auto-detect latest *.db in originalRecords)--update-conf — Update i3.conf community-webs.harvest.endpoint--skip-validate — Skip JSONL schema validation (not recommended)community-webs-ingest.sh:
--db=PATH — Pass to export script--skip-export — Use existing ZIP (endpoint must already point to it)--full — Run harvest + mapping + enrichment + jsonl--update-conf — Update i3.conf with export output directory$DPLA_DATA/community-webs/originalRecords/<YYYYMMDD>/community-webs-<timestamp>.zip$DPLA_DATA/community-webs/harvest/$DPLA_DATA/community-webs/The export script runs community-webs-validate-jsonl.py on the JSONL before zipping. It checks:
id"status":"deleted" are skipped (harvester filters these)Tests: ./venv/bin/python -m pytest scripts/tests/test_community_webs_validation.py -v
*.db in $DPLA_DATA/community-webs/originalRecords/ or use --db=/path/to/file.dbsqlite3, jq (brew install / apt install)community-webs.harvest.endpoint must point to directory containing the ZIP| Resource | Path |
|---|---|
| Script reference | scripts/SCRIPTS.md |
| Ingest docs | README_INGESTS.md |
| Agent guide | AGENTS.md |