context-graph-mining

name

context-graph-mining

description

INVOKE THIS SKILL when mining the procurement-agent Arize project for patterns in agent decisions vs human overrides, building a context graph, and proposing updates to procurement-agent rules. Use after a batch of reviews has been captured (process.run + override.run spans with annotations) to surface (a) where the agent's recommendations systematically diverge from reviewer overrides, (b) tribal knowledge embedded in reviewer reasoning that should become explicit rules, and (c) concrete code-change proposals against the procurement-agent codebase.

Context Graph Mining

This skill mines a procurement-agent Arize project for decision patterns, builds a context graph view of how the organization actually decides, and proposes concrete updates to the agent's rules and prompts.

When to use

• After a batch of human overrides has been applied (e.g. Vera Fye reviewing 100+ requests via procurement-reviewer/).
• When you want to surface rule-reality gaps — where the agent's encoded rules disagree with how the org actually decides.
• When you want concrete diff proposals against data/policies.json, data/vendors.json, or the evaluator system prompt.

What you have access to

• The procurement-agent Arize project — export traces, spans, and annotations via the arize-trace skill or the arize.exporter.ArizeExportClient Python SDK.
• The procurement-agent/ codebase — src/agent/evaluator.py, src/agent/tools.py, data/policies.json, data/vendors.json.
• The procurement-reviewer/src/reviewer.py system prompt — Vera Fye's institutional knowledge as currently encoded.

Workflow

Phase 1 — Discovery

Build the trace inventory.

For each session.id (one per purchase request):

• The process.run root span — agent's recommendation, confidence, reasoning, and tool calls (check_policy, lookup_vendor, check_budget).
• The override.run root span — the agent re-run with Vera's decision injected.
• Annotations on the override root: Reviewer Decision, Reviewer Confidence, Reviewer Name labels and the annotation.notes block (reasoning + precedent + conditions).

Use the arize-trace skill for shell-level exports. For programmatic export of annotations, use ArizeExportClient.export_model_to_df and read the annotation.<name>.label and annotation.notes columns.

Sanity check: confirm len(sessions) matches the expected request count and that every override span has a non-null Reviewer Decision label.

Phase 2 — Pattern discovery

Three priority queries surface the actionable patterns:

1. Rule-reality gap — every session where agent.recommendation != reviewer.decision. Group by (vendor, department, urgency, amount-band). Clusters with N >= 5 examples are candidates.
2. Precedent reuse — extract the precedent field from annotation.notes; cluster by similarity. Frequently-cited precedents are tribal knowledge that should become explicit rules.
3. Confidence-vs-override correlation — when agent.confidence == "high" but Vera overrides anyway, the agent has confidently wrong heuristics. Highest-priority targets.

Each pattern produces a structured finding:

"For requests matching X, the agent recommends Y but reviewers override to Z in N/M cases. Cited reason: '...'."

Include 2-3 example session IDs as evidence.

Phase 3 — Proposals

For each finding, read the relevant source and propose a concrete diff:

Finding type	Lands in
New approval rule	data/policies.json (and tool logic in src/agent/tools.py if behavior changes)
Vendor-specific signal	data/vendors.json
Prompt update	src/agent/evaluator.py system prompt
New tool	src/agent/tools.py + register on the agent

Output proposals as unified diffs with rationale, not as freeform prose.

Phase 4 — Validation handoff (out of scope here)

Validation runs as Arize experiments — out of scope for this skill. Just describe the experiment that would validate each proposal: input dataset (the captured override traces), metric (agreement-with-reviewer rate), regression check.

Output

Produce a single markdown report with this structure:

1. Summary — sessions analyzed, override rate, top three patterns.
2. Rule-reality gaps — table of clusters; for each: evidence (linked Arize sessions), proposed fix, expected effect.
3. Precedent extraction — top tribal knowledge found in annotation.notes.
4. Proposed diffs — unified diffs against the codebase, each with rationale and validation plan.
5. Open questions — patterns that need more data or are ambiguous.

Save the report to .context-graph-mining/report-<timestamp>.md (gitignored) and return its content as your final response.

Constraints

• Read-only — this skill does not apply diffs. Output is a markdown report; humans or downstream tooling apply changes.
• Cite evidence — every proposed change must reference at least 3 distinct Arize sessions. If a cluster has fewer, mark it as "weak signal" rather than a proposal.
• No fabricated patterns — if the data does not support a finding, say so.
• Domain-agnostic in spirit, procurement-specific in this run — the patterns and entities (vendor, department, policy) are derived from trace structure, not hardcoded.

Related skills

• arize-trace — export traces, spans, sessions.
• arize-annotation — read annotation configs and labels.
• arize-dataset / arize-experiment — for the validation handoff step described in Phase 4.
• arize-link — generate clickable links to evidence sessions for the report.