shadow-compare

name	shadow-compare
description	Compare Palimpsest shadow-mode output against legacy crawlers (Heritrix, Brozzler, Warcprox). Use during Phase 1 migration to validate artifact equivalence.
version	0.1.0
user-invocable	true
allowed-tools	Bash(cargo *) Bash(diff *) Bash(python3 *) Read Write Grep Glob
argument-hint	[palimpsest-dir] [legacy-dir]

Shadow Mode Comparison

Phase 1 of the migration runs Palimpsest alongside existing crawlers. This skill compares their outputs.

Parameters

• Palimpsest artifact directory: $0
• Legacy artifact directory: $1 (WARC files from Heritrix/Brozzler/Warcprox)

Comparison Dimensions

Coverage

• Same URLs captured (Palimpsest >= legacy)
• Same HTTP methods observed
• Redirect chains match

Content Fidelity

• Response bodies are byte-identical (after decompression normalization)
• Response headers captured (Palimpsest captures superset of legacy)
• Request headers captured (legacy often drops these)

Metadata Completeness

• Palimpsest envelopes contain DNS/TLS data legacy lacks
• Timing data available in Palimpsest but not legacy
• Resource graphs available in Palimpsest but not legacy

Format Compatibility

• Palimpsest WARC++ records parseable by standard WARC readers
• Extension records gracefully ignored by legacy tools

Output

Shadow Comparison Report
========================
Palimpsest artifacts: {n} ({size})
Legacy artifacts: {n} ({size})

Coverage:
  URLs in both: {n}
  Palimpsest only: {n} (list top 5)
  Legacy only: {n} (list top 5 — these are regressions)

Content fidelity:
  Byte-identical bodies: {n}/{total} ({pct}%)
  Header-identical: {n}/{total} ({pct}%)
  Divergent: {n} (list top 5 with diff summary)

Palimpsest advantages:
  With DNS snapshot: {n}
  With TLS fingerprint: {n}
  With resource graph: {n}
  With execution envelope: {n}

Verdict: {READY FOR PHASE 2 / NEEDS WORK}
Blockers: (list any regressions)