qualitative-analysis

name	qualitative-analysis
description	Fetch validated evidence bundles and event calendars, generate the grounded AI qualitative analysis artifact with one capped Claude API call, and validate and gate it before any rendering.

qualitative-analysis

The AI qualitative layer of AIMS (roadmap #98, contract #89): deterministic evidence and calendar inputs, one capped LLM call producing a schema-validated data/qualitative/<stem>.json artifact, and deterministic grounding gates. The quantitative pipeline stays the sole source of truth for scores, ranks, risk gates, and the market regime — this layer only explains or challenges them, and any failure here publishes the quantitative report unchanged.

Disclaimer: All output is informational only and does not constitute investment advice.

Scripts

All scripts are in .agents/skills/qualitative-analysis/scripts/ and delegate to src/aims/.

Script	Purpose
fetch_evidence.py	Build data/evidence/<stem>.json from yfinance news + macro feeds
validate_evidence.py	Validate an evidence bundle against data/schema/evidence.schema.json
update_calendars.py	Refresh data/calendars/earnings.json from yfinance earnings dates
validate_calendar.py	Validate a calendar file against data/schema/calendar.schema.json
qualitative_analysis.py	One Claude API call → validated, gated qualitative artifact
validate_qualitative.py	Validate a qualitative artifact (shape, enums, caps, grounding)
evaluate_stances.py	Deterministic stance-vs-forward-return evaluation (data/performance/)
validate_performance.py	Validate a stance-evaluation artifact
check_citation_links.py	Sample recent citation URLs; warn on link rot (never a gate)
prompt_regression.py	Structural regression harness for prompt/model changes
validate_prompt_regression.py	Validate data/performance/prompt_regressions.json

Usage

# 1. Deterministic inputs (no LLM, no API key needed)
uv run .agents/skills/qualitative-analysis/scripts/fetch_evidence.py \
    --analysis-date 2026-07-04 --output data/evidence/
uv run .agents/skills/qualitative-analysis/scripts/validate_evidence.py \
    --input data/evidence/2026-07-04.json

uv run .agents/skills/qualitative-analysis/scripts/update_calendars.py \
    --output data/calendars/earnings.json
uv run .agents/skills/qualitative-analysis/scripts/validate_calendar.py \
    --input data/calendars/earnings.json

# 2. The single non-deterministic step (requires ANTHROPIC_API_KEY)
ANTHROPIC_API_KEY=... \
uv run .agents/skills/qualitative-analysis/scripts/qualitative_analysis.py \
    --analysis data/analysis/2026-07-04.json \
    --evidence data/evidence/2026-07-04.json \
    --calendar data/calendars/macro_events.json \
    --calendar data/calendars/earnings.json \
    --output data/qualitative/

# 3. Validate the committed artifact (defense in depth)
uv run .agents/skills/qualitative-analysis/scripts/validate_qualitative.py \
    --input data/qualitative/2026-07-04.json \
    --analysis data/analysis/2026-07-04.json \
    --evidence data/evidence/2026-07-04.json

Contract highlights

• Scope: commentary covers the top-K published signals only (K=5, matching history.py) plus one market narrative and up to five themes — one LLM call per run, capped input and output.
• Grounding: the model may cite only evidence IDs from the day's committed bundle; the request schema constrains citations and canonical IDs at the API level, and validate_qualitative.py re-enforces everything regardless of what the model emits. Evidence text is untrusted data: instructions inside it are content to ignore.
• Machine-checkable claims: realized-price-action claims use direction_claim enums; every number in free text must be declared in numeric_claims. aims.qualitative_gates cross-checks the structured claims against the quantitative features and cited evidence — gated instrument entries are excluded from rendering, and a gated market narrative withholds the whole artifact (one regeneration retry first).
• Provenance: the prompt is the committed file prompts/qualitative_v1.md; its version and SHA-256, the model ID, and SHA-256 hashes of all inputs are recorded in artifact metadata. Changing the prompt or model requires the #97 regression harness (below).
• Fail-open: missing ANTHROPIC_API_KEY or an empty evidence bundle skips the step (exit 0); API errors or persistent validation failures exit non-zero without writing an artifact. The daily workflow treats all of these as non-fatal and publishes the quantitative report unchanged.

Stance evaluation (#97)

The daily workflow runs evaluate_stances.py after the qualitative-analysis step: it joins accumulated ungated stances with forward returns reconstructed purely from committed analysis artifacts, writes the validated data/performance/<date>.json artifact, and regenerates the public content/evaluation/_index.md page. Everything is deterministic and safe in the empty state (no qualitative artifacts yet → zero counts plus a warning). Results are informational association only — never presented as an investable track record.

Trust boundary: the qualitative-analysis step is continue-on-error, so a same-run data/qualitative/<stem>.json file can exist on disk even after a failed or unvalidated run. The workflow passes --exclude-qualitative-date "${DATE}" to evaluate_stances.py whenever steps.qualitative.outcome != 'success', so that date is excluded from evaluation (with a recorded warning) regardless of whether the file is present — the same trust boundary already used for report rendering and Slack notification (steps.qualitative.outcome == 'success'). Historical, already-validated qualitative artifacts are never affected.

check_citation_links.py additionally samples recent evidence URLs and logs link-rot warnings (never a gate).

Changing the prompt or model

Adoption is a reviewed human decision; there is no automatic prompt tuning. Before merging any change to prompts/qualitative_v1.md, to aims.qualitative.PROMPT_VERSION, or to aims.qualitative.MODEL_ID:

# 1. Generate candidate artifacts with the edited prompt/model from
#    committed inputs (or use the committed fixtures when no API key is
#    available), then run the harness and record the results:
uv run .agents/skills/qualitative-analysis/scripts/prompt_regression.py \
    --artifact tests/fixtures/qualitative_2024-01-01.json \
    --analysis tests/fixtures/analysis_2024-01-01_mapped.json \
    --evidence tests/fixtures/evidence_2024-01-01.json \
    --record data/performance/prompt_regressions.json

The harness asserts structural and gate metrics (validator cleanliness, market-narrative rendering, instrument gate pass rate ≥ 0.6, citation coverage = 1.0, stance-distribution sanity) — never exact prose — and records the run keyed by the prompt file's SHA-256 and the pinned model ID. Commit the updated prompt_regressions.json together with the change: a CI test fails when the committed prompt/model has no recorded passing entry.

Operational details (secrets, shadow mode, gate thresholds, costs, troubleshooting, recovery) are in OPERATIONS.md §10–§12.