Systematic workflow for finding, downloading, and indexing engineering literature by domain. Covers the full lifecycle: discovery via standards ledger and doc index, web search for open-access PDFs, download script generation, PDF validation, catalogue YAML creation, and handoff to the 7-phase document-index-pipeline for indexing. Use when populating a new engineering domain with reference literature or when a WRK item requires domain-specific standards and textbooks.
Systematic workflow for finding, downloading, and indexing engineering literature by domain. Covers the full lifecycle: discovery via standards ledger and doc index, web search for open-access PDFs, download script generation, PDF validation, catalogue YAML creation, and handoff to the 7-phase document-index-pipeline for indexing. Use when populating a new engineering domain with reference literature or when a WRK item requires domain-specific standards and textbooks.
Additional Domains (in standards but not yet in literature tree)
Domain
Typical Target Repo
catenary
digitalmodel
mooring
digitalmodel
risers
digitalmodel
drilling
OGManufacturing
bsee
worldenergydata
economics
worldenergydata
Legacy Standards Storage
The og_standards corpus lives at /mnt/ace/docs/_standards/ organized by org:
ABS, API, ASTM, BSI, DNV, ISO, MIL, NEMA, Norsok, OnePetro, Unknown.
Inventory DB: /mnt/ace/O&G-Standards/_inventory.db (SQLite, 6.8 GB).
Step-by-Step Procedure
Step 0 — Verify Domain Directory Exists
DOMAIN="geotechnical"# ← set your domain
LIT_DIR="/mnt/ace-data/digitalmodel/docs/domains/${DOMAIN}/literature"mkdir -p "${LIT_DIR}"ls -la "${LIT_DIR}"
If the domain is new, create standard subdirectories:
uv run --no-project python scripts/data/document-index/query-ledger.py \
--domain ${DOMAIN} --verbose
Record each standard's status: gap, done, wrk_captured, reference.
Standards with gap or wrk_captured are download candidates.
Step 2 — Query the Document Index
Search the 1M+ record index for existing documents in this domain:
uv run --no-project python -c "
import json
from collections import Counter
matches = []
with open('data/document-index/index.jsonl') as f:
for line in f:
rec = json.loads(line)
path_lower = rec.get('path', '').lower()
summary_lower = (rec.get('summary') or '').lower()
if '${DOMAIN}' in path_lower or '${DOMAIN}' in summary_lower:
matches.append(rec)
print(f'Found {len(matches)} documents')
by_source = Counter(r['source'] for r in matches)
for s, c in by_source.most_common():
print(f' {s}: {c}')
"
Prioritize og_standards and ace_standards sources — these are already local.
Step 3 — Cross-Reference Capability Map
Check what calculations exist vs. gaps in the target repo:
uv run --no-project python -c "
import yaml
with open('specs/capability-map/digitalmodel.yaml') as f:
data = yaml.safe_load(f)
for m in data['modules']:
if '${DOMAIN}' in m['module'].lower():
print(f\"Module: {m['module']} ({m.get('standards_count', '?')} standards)\")
for s in m.get('standards', [])[:30]:
print(f\" {s['status']:15s} {s['org']:8s} {s['id'][:70]}\")
"
Step 4 — Web Search for Open-Access Literature
Search for freely available PDFs across these source tiers:
Save script to: /mnt/ace-data/digitalmodel/docs/domains/${DOMAIN}/literature/download-literature.sh
Key script patterns:
Always source scripts/lib/download-helpers.sh for the download and log functions
Use set -uo pipefail (NOT set -e) — download failures should log, not abort
Guard fallible downloads with || true or || log "NOTE: ..."
Use descriptive filenames: Author-Year-Short-Title.pdf
The download function auto-skips existing files (resume-safe)
Always run --dry-run first to preview
Step 6 — Execute Downloads and Validate
# Dry run first
bash download-literature.sh --dry-run
# Execute
bash download-literature.sh
# Validate all PDFs are real PDFs (not HTML/WAF responses)
find "${LIT_DIR}" -name "*.pdf" -exec file {} \; | grep -v "PDF document"
Any file that shows "HTML document" or "ASCII text" instead of "PDF document"
is a WAF response. Move it to a _failed/ directory and add to pending_manual.
Save to specs/capability-map/research-briefs/${DOMAIN}.yaml:
uv run --no-project python scripts/data/research-literature/research-domain.py \
--category ${DOMAIN} --repo digitalmodel
Or produce manually following the template in the research-literature skill's
references/templates.md.
Step 9 — Trigger Document Index Pipeline
Hand off to the document-index-pipeline for indexing newly downloaded literature:
# Phase A — re-index to pick up new files
uv run --no-project python scripts/data/document-index/phase-a-index.py
# If full pipeline takes >5 min, queue it instead:echo"uv run --no-project python scripts/data/document-index/phase-a-index.py" \
> .Codex/work-queue/assets/${WRK_ID}/index-regen-queued.txt
The pipeline phases that follow:
Phase A: scan filesystem → index.jsonl
Phase B: LLM extraction + classification
Phase C: domain classification
Phase E: backpopulate index
See document-index-pipeline skill for full phase details.
Step 10 — Archive Dark Intelligence
For university coursework, worked examples, and methodology extractions:
mkdir -p knowledge/dark-intelligence/${DOMAIN}/
Save worked examples (problem statements + known answers) here for TDD test
generation. These are private resources — see dark-intelligence-workflow skill.
Pitfalls and Warnings
Download Pitfalls
WAF responses saved as PDF — Always validate with file *.pdf. Sites like
eagle.org (ABS) return HTML through Cloudflare WAF. The download helper saves
whatever it gets — you must check.
set -e kills download scripts — Use set -uo pipefail without -e.
The download function returns 1 on failure; with -e the script aborts on
the first 404. Guard individual downloads with || true.
Archive.org borrow-only — Some books show a download button but return 403.
Check the item page for "Borrow" vs "Download" before scripting.
Duplicate downloads across domains — Check if a standard already exists in
/mnt/ace/docs/_standards/ or another domain's literature before downloading.
Use the og_standards SQLite to search:
sqlite3 /mnt/ace/O&G-Standards/_inventory.db \
"SELECT path FROM documents WHERE path LIKE '%keyword%' LIMIT 20"
Large file timeouts — The download helper uses wget --timeout=60.
For large textbooks (>50 MB), increase timeout or download manually.
Indexing Pitfalls
pdfplumber hangs on NTFS/NFS — For batch PDF processing, use pdftotext
via subprocess, NOT pdfplumber. See WRK-1277 warning in document-index-pipeline.
Phase A must run before Phase B — New files won't be extracted/classified
until Phase A adds them to index.jsonl.
Budget awareness for Phase B — LLM extraction costs ~$0.002/doc (Haiku).
Large domain additions (100+ docs) should be batched and budget-tracked.
Catalogue Pitfalls
Every attempted resource must appear — Never silently skip a failed download.
It goes in pending_manual: with a reason (WAF, paywall, borrow-only, 404).
Validate local_path in YAML — Every local_path in the catalogue YAML
must point to an actually existing file. Script this check:
uv run --no-project python -c "
import yaml
from pathlib import Path
data = yaml.safe_load(open('knowledge/seeds/${DOMAIN}-resources.yaml'))
for section in ['textbooks', 'standards']:
for item in data.get(section, []):
p = Path(item.get('local_path', ''))
status = 'OK' if p.exists() else 'MISSING'
print(f' {status}: {p}')
"
AC Checklist
Domain literature directory exists with standard subdirectories
Standards ledger queried — gap standards identified
Document index searched — existing docs catalogued
