ingestion3

Workflow for generating and sending DPLA monthly pre-ingest emails to hub partners via AWS SES. Use this skill whenever someone asks to: send the monthly ingest email, prepare the DPLA pre-ingest notice, generate the hub email for a given month, preview the pre-ingest email, or do anything involving the pre-ingest email pipeline. Trigger even if the user just says something like "time to send the ingest email" or "it's email day" — context from the project makes the intent clear.

Run Community Webs ingest. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.

Remove DPLA items from the live site and search index in response to takedown requests. Use when the user says take down, delete, remove item(s) from dp.la, or action a takedown request. Accepts DPLA IDs directly or criteria (hub, institution, collection) to generate an ID list. Includes mandatory pre-flight verification, search-before-delete on Elasticsearch, and post-deletion confirmation with count delta verification.

Show key i3.conf config for a hub (provider, harvest.type, harvest.endpoint, schedule, email, setlist). Use when user asks for hub config, harvest type/endpoint, who gets emails, schedule months, or OAI setlist details.

Debug and fix DPLA hub ingestion failures (harvest/mapping/enrichment/jsonl/s3-sync/anomaly). Use when user asks why a hub failed, to debug an ingest failure, check an escalation report, or retry a failed hub/stage.

Show all active and completed ingests in one consolidated view. Use when user asks status of the ingests, ingest status, which hubs are running, show active ingests, or ingest progress.

Monitor a running IngestRemap (remap.sh / ingest.sh step 2) or orchestrator remap stages. Use when user asks whether mapping/enrichment/jsonl is done yet, to monitor remap progress, or to check which stage is currently running.

Generate/preview/draft/send the monthly pre-scheduling summary email to hub contacts scheduled for a month (from i3.conf schedule.months). Use when user asks to send the scheduling email, monthly scheduling email, notify hubs for a month, or pre-scheduling email.

Watch an OAI harvest log and report set-by-set progress + ETA (for hubs using harvest.setlist). Use when user asks to watch OAI harvest progress, track sets, estimate completion, or monitor a long OAI harvest.

Report which hubs have new JSONL staged in S3 for a given month, and optionally post the report to Slack. Use when user asks what hubs are staged/ready for indexing, /ingest staged, or what changed this month in S3.

Send an ingest summary email to a hub's configured contacts from i3.conf using the most recent (or specified) mapping output. Use when user asks send/resend a hub ingest summary email.

Run or monitor the DPLA Python ingest orchestrator. Use when the user says run orchestrator, parallel ingest, ingest status, run hubs, orchestrator dry-run, or retry failed hubs. Covers venv, main entry point, status script, and logs.

Execute a single-hub or manual ingest by following the correct runbook and scripts. Use when the user says run ingest for [hub], harvest [hub], remap [hub], or run the full pipeline for [hub]. Ensures harvest type is identified, correct runbook and scripts are used, and outputs are verified.

Run S3 sync, find latest S3 data, and perform AWS data operations for DPLA ingestion using the correct profile and scripts. Use when the user says sync to S3, check S3 sync, upload to S3, AWS bucket, check JSONL sync, latest S3 data for a hub, or when was hub last ingested.

Follow the project workflow when adding or modifying shell or Python scripts. Use when the user asks to add a script, create a new script, modify a script, or write a script for a task. Ensures POSIX bash, common.sh, documentation in SCRIPTS.md, and tests are created or updated and run.

Verify ingest outcomes and send failure or status notifications to Slack or tech@dp.la. Use when the user asks to verify the ingest, check if it succeeded, notify about a failure, or post to tech-alerts.

Show all active and completed ingests in one consolidated view. Use when user asks status of the ingests, ingest status, which hubs are running, show active ingests, or ingest progress.

Monitor a running IngestRemap (remap.sh / ingest.sh step 2) or orchestrator remap stages. Use when user asks whether mapping/enrichment/jsonl is done yet, to monitor remap progress, or to check which stage is currently running.

Watch an OAI harvest log and report set-by-set progress + ETA (for hubs using harvest.setlist). Use when user asks to watch OAI harvest progress, track sets, or estimate completion.

Report which hubs have new JSONL staged in S3 for a given month, and optionally post the report to Slack. Use when user asks what hubs are staged/ready for indexing or /ingest staged.

Run Community Webs ingest from SQLite DB. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.

Debug and fix DPLA hub ingestion failures (harvest/mapping/enrichment/jsonl/s3-sync/anomaly). Use when user asks why a hub failed, to debug an ingest failure, check an escalation report, or retry a failed hub/stage.

Show all active and completed ingests in one consolidated view. Use when user asks status of the ingests, ingest status, which hubs are running, show active ingests, or ingest progress.

Monitor a running IngestRemap (remap.sh / ingest.sh step 2) or orchestrator remap stages. Use when user asks whether mapping/enrichment/jsonl is done yet, to monitor remap progress, or to check which stage is currently running.

Generate/preview/draft/send the monthly pre-scheduling summary email to hub contacts scheduled for a month (from i3.conf schedule.months). Use when user asks to send the scheduling email, monthly scheduling email, notify hubs for a month, or pre-scheduling email.

Watch an OAI harvest log and report set-by-set progress + ETA (for hubs using harvest.setlist). Use when user asks to watch OAI harvest progress, track sets, estimate completion, or monitor a long OAI harvest.

Run or monitor the DPLA Python ingest orchestrator. Use when the user says run orchestrator, parallel ingest, ingest status, run hubs, orchestrator dry-run, or retry failed hubs. Covers venv, main entry point, status script, and logs.

Execute a single-hub or manual ingest by following the correct runbook and scripts. Use when the user says run ingest for [hub], harvest [hub], remap [hub], or run the full pipeline for [hub]. Ensures harvest type is identified, correct runbook and scripts are used, and outputs are verified.

Run S3 sync and AWS data operations for DPLA ingestion using the correct profile and scripts. Use when the user says sync to S3, check S3 sync, upload to S3, AWS bucket, or check JSONL sync.

Follow the project workflow when adding or modifying shell or Python scripts. Use when the user asks to add a script, create a new script, modify a script, or write a script for a task. Ensures POSIX bash, common.sh, documentation in SCRIPTS.md, and tests are created or updated and run.

Report which hubs have new JSONL staged in S3 for a given month, and optionally post the report to Slack. Use when user asks what hubs are staged/ready for indexing, /ingest staged, or what changed this month in S3.

Verify ingest outcomes and send failure or status notifications to Slack or tech@dp.la. Use when the user asks to verify the ingest, check if it succeeded, notify about a failure, or post to tech-alerts.