ingest-sparta

> STOP. READ THIS ENTIRE SKILL.MD BEFORE CALLING ANY ENDPOINT. # /ingest-sparta Thin skill wrapper around the real SPARTA pipeline at `${HOME}/workspace/experiments/sparta/`. ## What This Skill Does Dispatches commands to the real pipeline. Does NOT contain pipeline logic. ## What the Real Pipeline Does 40 numbered Python steps (00-18) that: 1. Fetch SPARTA-Data.xlsx + ATT&CK + CWE + D3FEND + NIST + ESA sources 2. Extract and normalize 2,147+ controls across 6 frameworks 3. Fetch ~10K URLs referenced by controls 4. Extract clean text from fetched HTML/PDF 5. Generate embeddings and knowledge anchors 6. Build relationship graph (deterministic + LLM-verified edges) 7. Generate QRAs with T0/T1.5/T2 quality cascade 8. Calibrate confidence thresholds via active learning 9. Export tiered graph Data stored in DuckDB (358MB production DB) + ArangoDB (via SpartaDataBridge). DuckDB migration to ArangoDB complete for pipeline steps (commit f8c5be4d). DuckDB remains as read-only data layer via SpartaDataBridge. ## Commands ```bash # Run a specific pipeline step (from sparta project root) cd ${HOME}/workspace/experiments/sparta uv run python -m sparta.pipeline.00_fetch_source uv run python -m sparta.pipeline.12_qra --run-id myrun --limit 100 uv run python -m sparta.pipeline.08_relationships --run-id myrun # Run tests uv run pytest tests/ -v uv run pytest tests/ -k "test_qra" # List available steps ls src/sparta/pipeline/[0-9]*.py ``` **Module path is `sparta.pipeline.NN_name`, NOT `sparta.pipeline.steps.NN_name`.** The old `steps/` subdirectory generation (56 files, e.g. `01d_map_cwe_nist`, steps 48-61) lives only on branch `feature/nrs-standardization` at `e1cba40`; on main its directory held nothing but stale bytecode and was removed 2026-08-11. Any `sparta.pipeline.steps.*` invocation fails with `No module named`. ## Pipeline Steps (real, in src/sparta/pipeline/) Verified against the filesystem 2026-08-11 (40 files, 00-18). Earlier revisions of this file listed steps 01b/01c/01d/01e, 05d, 11b, 12b/12c, 13b, 14-25, 48-55 and 60-61, none of which exist on main. | Step | Module | What It Does | |------|--------|-------------| | 00 | `00_fetch_source` | Download SPARTA xlsx from sparta.aerospace.org | | 01 | `01_extract_base` | Parse SPARTA spreadsheet into structured data | | 02a/02b | `02a_sheet_normalize`, `02b_sheet_audit` | Normalize and audit worksheets | | 03 | `03_sheet_enrich` | Enrich sheet records | | 04 | `04_context_merge`, `04b_child_bundle` | Merge context, bundle child records | | 05 | `05_url_inventory` | Build URL manifest from controls | | 06 | `06_fetch_urls`, `06b_alt_sources`, `06b_category_knowledge`, `06c_alt_sources_review`, `06c_fetch_cwe`, `06d_verify_alt_sources`, `06m_audit` | Fetch URLs, alternate sources, CWE | | 07 | `07_generate_knowledge_chunks`, `07_generate_pdf_html_chunks`, `07a_cwe_to_chunks`, `07b_generate_llm_knowledge_chunks`, `07c_audit`, `07c_build_knowledge_anchors` | Chunking and knowledge anchors | | 08 | `08_relationships`, `08c_cwe_summaries`, `08d_cwe_shortlist`, `08e_cwe_adjudicate` | Deterministic edges, CWE adjudication | | 09 | `09_relationships_llm`, `09a_relationships_llm_audit`, `09a_relationships_tune`, `09b_cc_conceptual`, `09r_reject_sample_audit` | LLM-verified edges | | 10 | `10_groupings` | Groupings | | 11 | `11_export_graph`, `11a_qra_inputs_audit` | Graph export, QRA input audit | | 12 | `12_qra` | QRA generation with quality cascade | | 13 | `13_persist_arango` | Persist to ArangoDB | | 14 | `14_embed_chunks` | Generate embeddings | | 15 | `15_map_cwe_nist` | CWE→NIST via MITRE Heimdall (recovered 2026-08-11; Rev4 caveat below) | | 16 | `16_qra_audit` | QRA quality audit | | 17 | `17_crosswalk_edges` | Deterministic crosswalk edges: ATLAS mitigates, CVE→CWE | | 18 | `18_ingest_security_corpora` | Dispatch to external ingests: atlas, kev, nvd, qa, refresh, epss, owasp_llm | Step-number prefixes are not unique: `06b`, `06c`, `07`, `07c` and `09a` each name two different modules. Always invoke the full module name. ## Framework Registry (what is actually in sparta_controls) Verified live 2026-08-11 after the framework alignment audit (active = `deprecated != true`). Every active framework except EMB3D participates in crosswalk edges; 419,326 total relationships (2026-08-12: 419,299 post-audit + 27 OWASP_LLM). | source_framework | Active controls | Loaded by | Edges | |------------------|---------------:|-----------|-------| | `NVD_CVE` | 372,716 | step 18 → `scripts/ingest_nvd_cvelist.py` | 154,604 CVE→CWE (step 17) | | `NIST` | 1,896 | numbered pipeline | 121,957 →SPARTA | | `ATT_CK_Enterprise` | 1,644 | numbered pipeline + domain relabel | yes | | `CISA_KEV` | 1,616 | step 18 → `scripts/backfill_kev_corpus.py` | 597 CVE→CWE (step 17) | | `SPARTA` | 1,110 | numbered pipeline | yes | | `CWE` | 944 | numbered pipeline | yes | | `CAPEC` | 615 | numbered pipeline | yes | | `D3FEND` | 424 | numbered pipeline | yes | | `ATT_CK_Mobile` | 316 | numbered pipeline + domain relabel | yes | | `MITRE_ATLAS` | 205 | step 18 → `scripts/ingest_atlas.py` | 246 mitigates (step 17) | | `EMB3D` | 170 | outside pipeline | none — mitigations only; threat nodes not yet ingested, so no edges are derivable (legitimate orphan) | | `ATT_CK_ICS` | 167 | numbered pipeline + domain relabel | yes | | `ESA` | 137 | numbered pipeline | 652 | | `ISO` | 23 | outside pipeline | yes | | `NASA` | 14 | outside pipeline | yes | | `OWASP_LLM` | 10 | step 18 → `scripts/ingest_owasp_llm.py` | 27 references_weakness_context → ATLAS/CAPEC/CWE | Retired labels (records survive with `deprecated: true` and provenance, never deleted): `NVD` (4,830 merged into `NVD_CVE` after identity analysis proved the same CVE record family), `ESA_Shield` (137 duplicates of ESA with `duplicate_of_control_id` pointers), `attack` (524 legacy techniques relabelled to their MITRE domain via STIX bundles; 12 quarantined — corrupt pipe-delimited ids and retired technique ids — deprecated), `retired_probe` (500 `nvdtest__` test residue). Adding a framework means TWO writes, not one: nodes into `sparta_controls` and edges into `sparta_relationships` (step 17 for deterministic families). Nodes alone are reachable by semantic and BM25 search but join no crosswalk chain. `monitor-sparta` check 30 (Framework Label Alignment) catches orphan frameworks and label drift; check 31 (Framework Label Congruence) catches edge labels that contradict their own endpoints. Use the exact label already present. Do not introduce a case or format variant of an existing framework. Known surfaced findings, deliberately not repaired: the `F36_SUPPLY_CHAIN` / `F36_THREAT_PROFILE` / `F36_SPARTA_EXPLORER` edge-only namespaces and the dangling endpoint references (F36/TA ids with no control documents). CVE records additionally carry EPSS enrichment FIELDS (`epss_score`, `epss_percentile`, `epss_model_version`, `epss_date`; 356,485 records = 95.2% of CVEs, refreshed via step 18 `epss`). EPSS rows are attributes of existing CVEs, never nodes. ## Crosswalk Paths (CWE → SPARTA) Edge counts verified against the live corpus 2026-08-11: | Path | Hops | Data Source | Edges (live) | |------|------|-------------|--------------| | CWE → SPARTA (direct) | 1 | SPARTA v3.1 `cwe_class_ids` | 2,825 | | NIST → SPARTA | 1 | numbered pipeline | 121,957 | | CAPEC → CWE | 1 | MITRE curated fields | 1,212 | ### Direct Path (SPARTA v3.1) Step 08 extracts `cwe_class_ids` from SPARTA Techniques and creates CWE→SPARTA edges. ```bash # Run Step 08 to create/update CWE→SPARTA edges uv run python -m sparta.pipeline.08_relationships --run-id <run-id> ``` ### CWE → NIST 2-hop Path (Heimdall): Step 15 The July pipeline rewrite dropped the old `steps/01d_map_cwe_nist` module; it was recovered onto main 2026-08-11 as `15_map_cwe_nist` (commit `0864539`). ```bash uv run python -m sparta.pipeline.15_map_cwe_nist --download --dry-run # preview uv run python -m sparta.pipeline.15_map_cwe_nist # apply ``` Populates `nist_control_ids` + `nist_source` on CWE controls from the MITRE Heimdall CSV (144 curated mappings; 138 CWEs present in the corpus enriched on recovery, 2-hop path verified live: `CWE-79 → SI-10 → 17 SV-* controls` via `tor_threats`). Note this writes a FIELD, not edges — CWE→NIST edge documents in `sparta_relationships` still need a relationships step to materialize the path for graph traversal. Provenance caveat (externally validated 2026-08-11): `mitre/heimdall_tools` is archived (last push 2022-02-05, superseded by SAF CLI) and the CSV maps to 800-53 **Rev 4** — hence `nist_source = "mitre_heimdall_800-53r4"` on every enriched doc. Translate through NIST's Rev4→Rev5 change analysis before materializing production edges; NIST warns those relationships are not one-to-one. ### Edge Casing in sparta_relationships Framework labels on edges are case-exact matches of `sparta_controls` labels since the 2026-08-11 `relationship_framework_label` repair (243,914 edges relabelled `nist`→`NIST`, `sparta`→`SPARTA`; zero lowercase labels remain). | Edge Type | source_framework | target_framework | |-----------|-----------------|------------------| | CWE→SPARTA | `"CWE"` | `"SPARTA"` | | NIST→SPARTA | `"NIST"` | `"SPARTA"` | | CAPEC→CWE | `"CAPEC"` | `"CWE"` | Write new edges with the exact `sparta_controls` label. monitor-sparta check 30 (`framework_label_alignment`) flags any edge whose framework label matches no control population. ## Explorer UX 8-tab React UI for SPARTA pipeline transparency. Launch via `./run.sh explorer`. - **Frontend**: Vite on `:3002` — renders `SpartaExplorer` with 8 view components - **API proxy**: Express on `:3001` — proxies `/api/memory/*` to daemon Unix socket - **Data**: ArangoDB via memory daemon. Verified 2026-08-11: 382,700 controls, 250,679 QRAs, 263,854 relationships, 6,854 URLs, 6,855 url_content rows Views: Overview, Sources, Controls, URLs, Knowledge, QRAs, Relationships, Pipeline. Keyboard: press 1-8 to switch tabs. Components at `pi-mono/packages/ux-lab/src/components/sparta/explorer/`. ## Related Skills | Skill | Relationship | |-------|-------------| | `/sparta-review` | Brandon persona assessment of QRA quality | | `/reality-check-sparta` | Adversarial data quality verification | | `/monitor-sparta` | Continuous T0/T1.5/T2 quality monitoring | | `/data-audit` | DuckDB coverage report (Control→URL→Knowledge→QRA) | | `/qra-review` | Human-in-the-loop QRA accept/reject TUI | | `/sparta-stress-test` | End-to-end query pipeline stress test | ## QRA Schema (CRITICAL) **Canonical field for control ID: `source_control_id`** | QRA Type | qra_type | source_control_id Example | Notes | |----------|----------|---------------------------|-------| | Native | `native` | `CAPEC-115`, `T1595`, `CWE-79` | Framework definition QRAs | | Relationship | `sparta_context` | Technique ID from relationship | SPARTA-linked QRAs | **Legacy field:** `control_id` exists in older QRAs. Code queries BOTH fields: ```python cid = qra.get("source_control_id") or qra.get("control_id") or "" ``` **When querying by framework:** ```aql // Native QRAs (from /create-qras skill) FOR q IN sparta_qra FILTER STARTS_WITH(q.source_control_id, "CAPEC-") RETURN q // BOTH native and legacy FOR q IN sparta_qra LET cid = q.source_control_id != null ? q.source_control_id : q.control_id FILTER STARTS_WITH(cid, "CWE-") RETURN q ``` ## Environment ```bash SPARTA_ROOT=${HOME}/workspace/experiments/sparta # auto-detected # DuckDB: data/runs/<run-id>/sparta.duckdb # ArangoDB: via SpartaDataBridge (graph_memory) # Embedding: port 8602 ```