| name | context-graph-mining |
| description | INVOKE THIS SKILL when mining the procurement-agent Arize project for patterns in agent decisions vs human overrides, building a context graph, and proposing updates to procurement-agent rules. Use after a batch of reviews has been captured (process.run + override.run spans with annotations) to surface (a) where the agent's recommendations systematically diverge from reviewer overrides, (b) tribal knowledge embedded in reviewer reasoning that should become explicit rules, and (c) concrete code-change proposals against the procurement-agent codebase. |
Context Graph Mining
This skill mines a procurement-agent Arize project for decision patterns, builds a context graph view of how the organization actually decides, and proposes concrete updates to the agent's rules and prompts.
When to use
- After a batch of human overrides has been applied (e.g. Vera Fye reviewing 100+ requests via
procurement-reviewer/).
- When you want to surface rule-reality gaps — where the agent's encoded rules disagree with how the org actually decides.
- When you want concrete diff proposals against
data/policies.json, data/vendors.json, or the evaluator system prompt.
What you have access to
- The
procurement-agent Arize project — export traces, spans, and annotations via the arize-trace skill or the arize.exporter.ArizeExportClient Python SDK.
- The
procurement-agent/ codebase — src/agent/evaluator.py, src/agent/tools.py, data/policies.json, data/vendors.json.
- The
procurement-reviewer/src/reviewer.py system prompt — Vera Fye's institutional knowledge as currently encoded.
Workflow
Phase 1 — Discovery
Build the trace inventory.
For each session.id (one per purchase request):
- The
process.run root span — agent's recommendation, confidence, reasoning, and tool calls (check_policy, lookup_vendor, check_budget).
- The
override.run root span — the agent re-run with Vera's decision injected.
- Annotations on the override root:
Reviewer Decision, Reviewer Confidence, Reviewer Name labels and the annotation.notes block (reasoning + precedent + conditions).
Use the arize-trace skill for shell-level exports. For programmatic export of annotations, use ArizeExportClient.export_model_to_df and read the annotation.<name>.label and annotation.notes columns.
Sanity check: confirm len(sessions) matches the expected request count and that every override span has a non-null Reviewer Decision label.
Phase 2 — Pattern discovery
Three priority queries surface the actionable patterns:
- Rule-reality gap — every session where
agent.recommendation != reviewer.decision. Group by (vendor, department, urgency, amount-band). Clusters with N >= 5 examples are candidates.
- Precedent reuse — extract the
precedent field from annotation.notes; cluster by similarity. Frequently-cited precedents are tribal knowledge that should become explicit rules.
- Confidence-vs-override correlation — when
agent.confidence == "high" but Vera overrides anyway, the agent has confidently wrong heuristics. Highest-priority targets.
Each pattern produces a structured finding:
"For requests matching X, the agent recommends Y but reviewers override to Z in N/M cases. Cited reason: '...'."
Include 2-3 example session IDs as evidence.
Phase 3 — Proposals
For each finding, read the relevant source and propose a concrete diff:
| Finding type | Lands in |
|---|
| New approval rule | data/policies.json (and tool logic in src/agent/tools.py if behavior changes) |
| Vendor-specific signal | data/vendors.json |
| Prompt update | src/agent/evaluator.py system prompt |
| New tool | src/agent/tools.py + register on the agent |
Output proposals as unified diffs with rationale, not as freeform prose.
Phase 4 — Validation handoff (out of scope here)
Validation runs as Arize experiments — out of scope for this skill. Just describe the experiment that would validate each proposal: input dataset (the captured override traces), metric (agreement-with-reviewer rate), regression check.
Output
Produce a single markdown report with this structure:
- Summary — sessions analyzed, override rate, top three patterns.
- Rule-reality gaps — table of clusters; for each: evidence (linked Arize sessions), proposed fix, expected effect.
- Precedent extraction — top tribal knowledge found in
annotation.notes.
- Proposed diffs — unified diffs against the codebase, each with rationale and validation plan.
- Open questions — patterns that need more data or are ambiguous.
Save the report to .context-graph-mining/report-<timestamp>.md (gitignored) and return its content as your final response.
Constraints
- Read-only — this skill does not apply diffs. Output is a markdown report; humans or downstream tooling apply changes.
- Cite evidence — every proposed change must reference at least 3 distinct Arize sessions. If a cluster has fewer, mark it as "weak signal" rather than a proposal.
- No fabricated patterns — if the data does not support a finding, say so.
- Domain-agnostic in spirit, procurement-specific in this run — the patterns and entities (vendor, department, policy) are derived from trace structure, not hardcoded.
Related skills
arize-trace — export traces, spans, sessions.arize-annotation — read annotation configs and labels.arize-dataset / arize-experiment — for the validation handoff step described in Phase 4.arize-link — generate clickable links to evidence sessions for the report.
