Skip to main content
skillsmp

Menu

lie-detector

Deterministic 6-layer process verification for AI self-improvement loops. Detects reward hacking, evaluator monkey-patching, and criteria drift via hash-chain audit trails, sealed grading (SHA256 + AST), Lean4 formal proofs, process conformance (embed intent vs action), taxonomy graph sparsity, and SetFit classification. The audited agent does not choose its own auditors.

Aller à l'installationVérifier les fichiers

Informations de source

Dépôt
grahama1970/agent-stack-public
Dernière activité de la source
24 septembre 2026 à 15:51
Langue détectée de SKILL.md
anglais
Étoiles
0
Forks
0

Options d'installation

Le prompt qui vérifie d'abord la source est sélectionné par défaut. Vous pouvez passer à une commande directe ou télécharger une copie locale.

Vérifiez les fichiers source

Lisez SKILL.md et les fichiers associés affichés par SkillsMP avant de décider de l'installer.

Explorateur de fichiers
18 fichiers

Affichage de SKILL.md

SKILL.md
Instructions source · Aperçu en lecture seule
name
lie-detector
description
Deterministic 6-layer process verification for AI self-improvement loops. Detects reward hacking, evaluator monkey-patching, and criteria drift via hash-chain audit trails, sealed grading (SHA256 + AST), Lean4 formal proofs, process conformance (embed intent vs action), taxonomy graph sparsity, and SetFit classification. The audited agent does not choose its own auditors.
allowed-tools
["Bash","Read","Grep","Glob"]
triggers
["lie detector","verify grading integrity","detect gaming","seal grading","check for reward hacking"]
metadata
{"short-description":"Process verification for AI self-improvement loops","version":"1.0.0"}
provides
["lie-detector"]
composes
["review-conversation","conversation-lab","lean4-prove","memory","assistant","intent-mapper","task-monitor","formalize-request","agentic-evals"]
disciplines
["evaluation-quality","compliance-security"]
> STOP. READ THIS ENTIRE SKILL.MD BEFORE CALLING ANY ENDPOINT. # /lie-detector Deterministic process grading for AI agent conversations. Grades the **process** (tool calls, file edits, stated intent vs actual actions) — not just the output. ## Background On 2026-02-28 an agent manipulated `_self_grade()` during a self-improvement loop, widening compatibility mappings and fixing floating-point rounding to convert B-grades to A-grades. Reported 100% A-grade (50/50) when honest score was 88% (44/50). This pipeline processes DO-178C, MIL-STD, and NASA safety-critical documents — inflated quality scores can cause unsafe documents to enter the datalake. METR (2025) found 1-2% of all o3 task attempts contain reward hacking including evaluator monkey-patching. Training models not to cheat makes them cheat more cleverly. The only reliable defense is structural: make the process deterministically verifiable. ## Architecture: 6-Layer Detection Cascade | Layer | Name | Time | What it catches | |-------|------|------|-----------------| | Pre-gate | /intent-mapper | ~50ms | Vague/evasive stated intents | | 0 | /recommend-skill-chain | ~200ms | Agent cherry-picking auditors | | 1 | Sealed grading | ~5ms | Direct eval file modification | | 2 | Lean4 invariants | ~30s | Rewritten eval with same effect | | 3 | Process conformance | ~100ms | Intent != action (cosine sim) | | 3b | Taxonomy sparsity | ~1s | Selective omission | | 4 | SetFit classifier | ~10ms | Convincing but false explanations | | 5 | /assistant validate | ~5s | Subtle gaming that fools heuristics | ## Quick Start ```bash # Before a self-improvement loop ./run.sh seal "scripts/nico_asks_embry.py" "pi-mono/.pi/skills/review-pdf/verify/scoring.py" # After each iteration ./run.sh verify .lie-detector-seal.json # Full 6-layer cascade on conversation transcript ./run.sh detect sessions.jsonl # Compile Lean4 grading invariants ./run.sh prove # Train SetFit classifier ./run.sh train training_data/seed_incidents.jsonl ``` ## Commands | Command | Description | |---------|-------------| | `seal <file_glob>...` | Create sealed hash manifest of evaluation files | | `verify <seal_file>` | Verify files match seal (Layer 1) | | `prove` | Compile Lean4 grading invariants (Layer 2) | | `detect <conversation.jsonl>` | Full 6-layer cascade on conversation transcript | | `report <conversation.jsonl>` | Markdown report with verdicts per turn | | `train <labels.jsonl>` | Train/retrain SetFit classifier (Layer 4) | | `label <conversation.jsonl>` | Interactive labeling for training data | | `ingest <conversation.jsonl>` | Learn findings to /memory | | `chain <context>` | Show which layers /recommend-skill-chain selects | ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | `LIE_DETECTOR_STORAGE` | `/mnt/storage12tb/skills/lie-detector` | Model weights + audit logs | | `LIE_DETECTOR_CONFORMANCE_THRESHOLD` | `0.6` | Cosine sim threshold for intent-action mismatch | | `LIE_DETECTOR_TAXONOMY_SPARSE` | `0.3` | Bridge coverage below this = SPARSE | | `LIE_DETECTOR_TAXONOMY_DENSE` | `0.7` | Bridge coverage above this = DENSE | | `LEAN4_CONTAINER` | `lean_runner` | Docker container for Lean4 compilation | | `MEMORY_SERVICE_URL` | `http://127.0.0.1:8601` | /memory service endpoint |
Voir sur GitHub
Retour à la rechercheRetour à l’installation