legal-doc-processor

name

legal-doc-processor

description

Convert legal document collections into AI-ready JSONL datasets using the bundled legal-doc-processor project. Use when Codex needs to run or adapt the legal-doc-processor workflow for PDF, DOCX, PPTX, EML, MSG, or TXT legal files; classify documents as caselaw, published, private, or uncertain; remove PII from private documents; create RAG chunks and fine-tuning records; generate provenance manifests; review PII flags; or run adversarial re-identification and second-pass redaction.

Legal Document Processor

Use the bundled processor at assets/processor-source/ to turn legal document folders into structured JSONL datasets for RAG retrieval and fine-tuning. Keep processing local unless the user explicitly asks to use a remote service; legal documents may contain privileged, confidential, or regulated information.

Workflow

1. Inspect the input folder and confirm the output path. Prefer a dry run first for any new collection.
2. Check setup from assets/processor-source/requirements.txt. Use python3.12 explicitly when available.
3. Run classification without writing output:

   cd /path/to/skill/assets/processor-source
   python3.12 process.py --input /path/to/docs --output /path/to/output --dry-run
   

4. Review surprising classifications with the user before a full run, especially for private/client files.
5. Run the full processor:

   python3.12 process.py --input /path/to/docs --output /path/to/output
   

6. For larger collections, use --workers N. For provenance prefill, pass --sidecar sources.csv.
7. After processing, review output/provenance.json, output/review/review_log.jsonl, and output/review/ocr_queue/.

Bundled Processor

The complete upstream project is stored in assets/processor-source/. Read files there as needed rather than copying long implementation details into context.

Key entry points:

• process.py: main CLI for document processing.
• review_server.py: local browser review server for PII decisions.
• apply_decisions.py: apply a decisions CSV and write audit entries.
• re_id_risk.py: Anthropic-powered adversarial re-identification assessment.
• second_pass.py: targeted second-pass redaction using red-team findings or blocklists.
• tests/: pytest coverage for core pipeline behavior.

For a compact behavior map, read references/processor-guide.md.

Commands

Install runtime dependencies:

cd /path/to/skill/assets/processor-source
pip3 install -r requirements.txt
python3.12 -m spacy download en_core_web_lg

Run with a copyright/source sidecar:

python3.12 process.py --input /path/to/docs --output /path/to/output --sidecar sources.csv

Start interactive PII review:

python3.12 review_server.py --output /path/to/output

Apply review decisions from CSV:

python3.12 apply_decisions.py --decisions decisions.csv --output /path/to/output --reviewer "Reviewer Name"

Run re-identification risk assessment only when the user has provided or approved ANTHROPIC_API_KEY use:

ANTHROPIC_API_KEY=... python3.12 re_id_risk.py --output /path/to/output --apply

Preview and apply second-pass redaction:

python3.12 second_pass.py --output /path/to/output --dry-run
python3.12 second_pass.py --output /path/to/output

Output Checks

Confirm these outputs exist after a full run:

• rag/: JSONL chunks for caselaw and published sources.
• finetune/dataset.jsonl: fine-tuning records for private, uncertain, and published documents.
• review/review_log.jsonl: medium-confidence PII flags requiring human review.
• review/ocr_queue/: scanned PDFs with low OCR confidence.
• summary.html: human-readable processing and review report.
• provenance.json: hashes, processing metadata, source/copyright fields, and audit trail.

Safety

Treat uncertain documents as private and PII-bearing. Do not claim the generated dataset is legally safe, privileged-safe, HIPAA-safe, or sale-ready merely because the processor completed. Always surface remaining review steps, OCR failures, provenance blanks, and re-identification findings.