revolving-door-tracker

name	revolving-door-tracker
description	Map lobbying influence networks, parse government roles, audit financial data quality, query campaign donations, detect alumni clusters, apply OSINT heuristics, calculate revolving door metrics, track lobbying relationship filing spans and terminations, generate executive dashboards, and export interactive network visualizations
license	MIT
version	1.1.0
metadata	{"risk":"none","date_added":"2026-06-03"}

Revolving Door Tracker — All Milestones Complete (M1-M65, G1-G14, DQ1-DQ17, A1-A7)

Builds deterministic crosswalks between Senate LDA JSON filings and House LDA XML filings, resolves and deduplicates lobbying entities, normalizes the Congress members roster, parses government covered positions (16 regex pattern groups + a member-first compound-position pre-check + heuristic + LLM fallback), constructs a heterogeneous influence graph (~56K nodes, ~1.42M edges, temporal filing spans and LDA termination flags per lobbying edge), audits financial data-quality, matches campaign contributions (corporate PAC & individual FEC donor with nickname expansion), runs temporal analysis and alumni cluster detection, applies transnational OSINT alerts (including reverse revolving door signals), generates standalone interactive Vis.js visualizers with timeline slider and cluster highlighting, produces executive summary dashboards, and provides a Rich-formatted CLI REPL for investigative journalists.

When to Use This Skill

• When aligning Senate and House Lobbying Disclosure Act (LDA) data for congressional investigation.
• When generating clean organization and lobbyist entity tables from noisy, self-reported public records.
• When parsing government covered positions from free-text strings using regular expressions, heuristics, and LLM fallback.
• When auditing financial filings for income or expense reporting omissions according to self-filer and third-party rules.
• When mapping campaign contributions (LD-203 & individual FEC donations) to congressional politicians.
• When building a heterogeneous network graph to identify revolving-door anomalies.
• When tracking lobbying intensity temporal trends and identifying ex-staffer alumni networks (clusters).
• When checking whether a lobbyist-client relationship was still active, terminated, or dormant within the corpus window, or whether a lobbyist may have returned to government (reverse revolving door).
• When investigating transnational connections, Delaware shell entities, shared addresses (co-location), and foreign government influence.
• When generating browser-based standalone Vis.js network visualizers with timeline filtering and cluster highlighting.
• When producing executive summary dashboards with metric cards, score histograms, and issue heatmaps.
• When running interactive CLI investigator sessions (REPL) for journalistic inquiry with Rich-formatted output.
• When exporting network subgraphs as GraphML for analysis in Gephi, Cytoscape, or NetworkX.
• When generating GAIN Agent Challenge interaction traces for session auditability.

Core Capabilities

• XML Safety: Parses House XML disclosures securely via defusedxml + lxml.etree with no_network, resolve_entities=False, load_dtd=False, huge_tree=False flags and 50MB size limit to prevent XXE, DTD expansion, and XML bomb attacks.
• Deterministic Crosswalk: Connects House disclosures with Senate records achieving >=99.90% coverage. Includes length ratio guard (0.78), "ON BEHALF OF" suffix comparison, and fuzzy organization matching with three-way Levenshtein strategy.
• Lobbying Entity Resolution: Deduplicates registrants, clients, and lobbyists using fuzzy matching (>=0.90 threshold), token_sort_ratio fallback for inverted names, and cross-registrant deduplication (0.95 threshold) with 2-char prefix blocking for O(n·k) performance (M58).
• Covered Position Parser: 3-layer parser with 16 regex pattern groups (Chief of Staff, Legislative Director, Press Secretary, Staff Assistant, Professional Staff Member, District Director, Scheduler, Detailee/Fellow, Budget Analyst, etc.), heuristic fallback, and optional LLM Layer 3 (claude-haiku-4-5-20251001). A member-first compound-position pre-check (A1) resolves "Senator <Name>, <role>..." strings to the role after the comma instead of mislabeling the staffer as Member of Congress. Per-layer statistics logged for observability.
• Financial Quality Auditor: Flags third-party registrants omitting income and self-filers omitting expenses. Reports include summary section with total_filings_analyzed, omission_rate, and explanatory notes.
• FEC Campaign Integration: Queries individual contributor donations via FEC API with 45s timeout (M60), exponential backoff (2s, 4s), circuit breaker (3 failures → 120s cooldown with auto-reset), disk cache (fec_cache.json), and elapsed-time response logging. Each donation amount uses the individual contribution_receipt_amount (not the cumulative contributor_aggregate_ytd); with per_page=10, total_amount is the sum of the up to 10 most recent receipts — not a lifetime total — and it does not feed the anomaly score. The query filters by contributor name only, so records may include homonyms: every fec_enrichment carries identity_verified: false + a caveat, and lead sheets/visualizer show a "NOT identity-verified" warning (DQ17). Without API key: fields are left empty ("FEC data unavailable"). No data is fabricated or simulated.
• Contribution Matching: Enhanced match_member() with token_sort_ratio fallback for inverted names and nickname expansion dictionary (~55 entries: Bob→Robert, Chuck→Charles, etc.) achieving ~65-70% resolution rate. The graph builder's match_member() also resolves a single-token principal (bare surname, e.g. "Murray" from "Chief of Staff Senator Murray") to the sole roster member with that last name when unambiguous, collapsing the duplicate GovernmentEntity onto the Member node instead of a fragmented worked_for edge; ambiguous surnames are left unresolved.
• Person Name Deduplication: Strips middle initials and trailing single-letter tokens via normalize_person_name(); truncated-name merge pass with registrant guard prevents cross-org false merges (DQ13). Candidate dedup by org_id eliminates same-registrant duplicates; nickname expansion in cross-registrant dedup Phase 1.5; counter-finding thematic verification against COMMITTEE_ISSUES mapping; cluster size display as "N leads of M" (DQ14).
• Homonym Middle-Initial Guard (DQ17): middle_initials_compatible() blocks merging two names with different present middle initials (John A Anderson ≠ John T Anderson) while still merging absent-vs-present (Susan Zook ↔ Susan S Zook, DQ13). A no-middle variant compatible with several lettered homonyms is attached to the lettered cluster it shares a registrant firm with (via ResolvedLobbyist.registrant_names) when exactly one matches, else kept isolated. Applied across the exact/nickname/fuzzy merge passes and the intra-registrant match; normalize_person_name() is unchanged (M58 blocking + DQ13 intact).
• Circuit-Tied Registrant (DQ17): In scoring, current_registrant resolves from the registrant filed alongside the scored client (client + registrant lobbies_for edges share a filing id), not an arbitrary node-level registrant_id — correct for lobbyists active at multiple firms; falls back to the node id then the client name.
• Temporal Analysis: Evaluates quarterly and annual lobbying activity trends (increasing, decreasing, stable).
• Relationship Filing Spans (A2-A3): Lobbying edges carry filing_first/filing_last ("YYYY-QN" stamps, min/max-merged across deduplicated filings) and has_termination (LDA termination type codes: quarter code with T/@ suffix). Each candidate records filing_span and a relationship_status — active (filed within the corpus's latest quarters, one-quarter filing latency tolerated), terminated (LDA termination filing observed), or not_recent — rendered as "Filings Observed"/"Relationship Status" in FINDINGS.md and lead sheets, so historical relationships are never mistaken for current ones.
• Reverse Revolving Door Alerts (A4): When a lobbyist's most recent filing across all their relationships lags the corpus max by >= 2 quarters, the case carries an OSINT alert (they may have left lobbying or returned to government service). Informational only — no score impact.
• Name Variants & Display Casing (A5-A6): Cases list every genuinely different spelling of the person's name found in filings ("Also Appears in Filings As", case-only noise filtered); display_name() re-cases the Mc surname prefix for presentation while normalize_name() stays frozen (it feeds person-key hashes and the FEC cache key). Registrant/client names reproduce the official LDA filing spelling — documented in the FINDINGS methodology notes.
• Cluster Detection: Identifies networks of ex-staffer alumni working for the same firm or lobbying the same client.
• Defensive Counter-Findings: Formulates defensive arguments (cooling-off compliance, non-overlapping committees) for journalistic auditability.
• Transnational & OSINT Due Diligence: Checks for Delaware incorporations, common registry addresses (co-location), and foreign government financing.
• Scoring Engine: Two-circuit scoring (Circuit A direct + Circuit B committee-based) with configurable weight (default 1.5). Formula: base 3.0 + committee-issue alignment (0 to +2, scaled by the fraction of the client's aligned issues) + temporal proximity within 2 years inclusive (+1) + high lobbying volume (+1, corpus-wide P75) + press mentions (+1) + multi-circuit activity (+1) + contribution amount tier (0 to +1: +1 if >= $15k, +0.5 if >= $5k) + Circuit B bonus (default +1.5, specific non-broad issue codes only), clamped to 10.0. A separate high/medium/low confidence flag reflects data quality independently of the score.
• Interactive Investigator CLI Agent: Rich-formatted REPL with colored tables, styled panels, score severity coloring (red >= 7, yellow >= 5, green < 5). Commands: status, list-leads, investigate, compare, add-hypothesis, discard-lead, fec-search. Session traces auto-saved to interaction_traces.json.
• Visualizer Generator: Extracts candidates with score >= 6.0 and builds standalone Vis.js HTML with custom glassmorphism tooltip cards (G1), XSS-hardened innerHTML via escapeHtml() + event delegation (G3), pinned vis-network CDN @9.1.9 (G4), loading overlay with spinner (G5), keyboard shortcuts (Esc/F/P), timeline slider (Q1–Q4 quarterly filter), alumni cluster highlighting (pulsing golden border), GraphML/PNG/CSV export, edge label and amount display fix in Direct Connections (G7), lead score filter slider 0–10 (G8), ego network isolation via Focus Network button (G9), 7 edge type filter checkboxes (G10), collapsible sorted leads list panel (G11), leads list dedup by person name (G12), dynamic score slider minimum (G13), and timeline edge count feedback (G14).
• Executive Dashboard: Standalone HTML with metric cards, top-10 candidates table, SVG score distribution histogram, and LDA issue code heatmap.
• Pipeline Performance: Senate filings cache (~25s/run savings), incremental NER cache v2 (per-file hashing with NER-only entity hash for M57, <1 min on reruns vs ~81 min full), MessagePack binary graph I/O via graph_io.py for ~3-5x faster serialization (M59), and parallel stage execution (Batch A: contributions + quality audit; Batch B: visualizer + dashboard + findings report).
• Pipeline Determinism: All outputs produce identical SHA-256 hashes across runs. Strict sorting before serialization, no list(set()) without sorted().
• Progress Bars: Real-time Rich progress bars with percentage, elapsed time, and ETA in stages 2, 3, 4, and 6. Additional progress indicators for cross-registrant dedup, members roster building, coverage report generation, and entities.json serialization (M52). Graceful fallback to standard logging if Rich unavailable.
• Contribution Resolution Logging: Logs unique names resolved vs. items resolved by cache reuse, clarifying that ~89K of 92K resolved items come from repeated honoree names (M53).
• FEC API Summary Logging: After FEC lookups, logs a summary with cache hits, API successes, timeouts, and errors for at-a-glance diagnostics (M54).
• NER Parallel Processing: --ner-workers N CLI flag for 07_press_ner.py enables SpaCy nlp.pipe(n_process=N) parallelization on Linux/macOS. On Windows, uses ProcessPoolExecutor for chunk-level parallelization (M61). ~85 min → ~25 min with 4 workers.
• Per-Stage Elapsed Time: All 12 pipeline stage scripts log their elapsed time at completion via time.monotonic() for performance profiling (M56).
• NER Entity Hash Refinement: Cache hash computed only from NER-relevant fields (entity_type, name, raw_names), so position resolution in stage 03 does not invalidate the cache. Saves ~86 min/run on warm cache (M57).
• Cross-Registrant Dedup Blocking: 2-character prefix blocking reduces fuzzy match comparisons from O(n²) to O(n·k). Lossless at 0.95 threshold. Saves ~10 min/run (M58).
• MessagePack Graph I/O: graph_io.py provides load_graph()/save_graph() with automatic format detection by extension (.msgpack vs .json). Default output: graph.msgpack. ~3-5x faster than JSON. Saves ~5 min/run (M59).
• FEC API Timeout Tuning: Timeout raised to 45s, circuit breaker cooldown raised to 120s, with elapsed-time debug logging on successful responses (M60).

Reproducibility

All pipeline outputs are deterministic given the same input corpus and API caches. Two external APIs (Anthropic LLM and FEC) introduce variance on fresh runs:

• Reference caches are shipped in reference_outputs/ (resolved_positions.json + fec_cache.json). The pipeline runner auto-copies them to checkpoints/ on first run, ensuring identical outputs without API keys.
• Without caches: regex + heuristic covers ~95% of position parsing; FEC individual contribution fields are left empty ("FEC data unavailable"). No data is fabricated. Pipeline completes successfully but scores may differ.
• Determinism verification: test_determinism.py checks SHA-256 hashes of all checkpoint files across runs.

Related Artifacts

• Findings Report: FINDINGS.md — Auto-generated by 13_generate_findings.py (Stage 12). 15 curated revolving door cases with provenance, supported by entity resolution (M1-M5), scoring (M13), FEC enrichment (M21), cluster detection (M22), and press NER integration (M23). Regenerated on each pipeline run.
• Interaction Traces: interaction_traces.json — Session logs from pipeline execution and REPL investigator sessions, keyed to skill invocations.
• Investigation Guide: investigation_guide.md — Non-technical handbook for journalists analyzing leads.