codex-autoresearch-skill

# Codex Autoresearch Skill > Skill by [ara.so](https://ara.so) — Codex Skills collection. ## What is Codex Autoresearch? Codex Autoresearch is an autonomous goal-driven experimentation system for Codex that continuously cycles through: modify code → verify result → retain (if improved) or discard (if worse) → repeat indefinitely. Tell Codex what you want to improve, walk away, and come back to a log of experiments and a better codebase. **Key capabilities:** - Autonomous iteration loops (foreground or background) - Git-based experiment tracking with automatic revert on failure - Dual-gate verification (did it improve? did anything break?) - Escalating retry strategies (REFINE → PIVOT → Web search → Stop) - Cross-run learning from past experiments - Multiple modes: loop, plan, debug, fix, security, ship, exec ## Installation ### Recommended: Skill Installer ```bash # In Codex $skill-installer install https://github.com/leo-lilinxiao/codex-autoresearch ``` Then restart Codex. ### Manual Installation ```bash # Clone to Codex skills directory cd ~/.config/codex/skills git clone https://github.com/leo-lilinxiao/codex-autoresearch # Or create a symlink ln -s /path/to/codex-autoresearch ~/.config/codex/skills/codex-autoresearch ``` ## Prerequisites for Full Functionality **For background runs and unattended execution:** ```bash # Start Codex with Goals, hooks, and Full Access enabled codex --enable goals --enable hooks --dangerously-bypass-approvals-and-sandbox ``` **For foreground-only usage:** Standard Codex workspace permissions are sufficient. ## Core Command ```bash $codex-autoresearch ``` This is the single entry point. Codex will infer the mode and configuration from your natural language goal. ## Basic Usage Patterns ### 1. Improve Test Coverage ```text You: $codex-autoresearch I want to improve test coverage to at least 80% Codex: Current coverage: 62% (measured via pytest --cov) Target: ≥80% Verify: pytest --cov --cov-report=term Guard: pytest passes Results directory: ./autoresearch-results/ Run mode: foreground or background? You: Foreground, go. Codex: Starting iteration loop — baseline: 62% ``` **What happens:** - Codex identifies untested files/functions - Writes new tests or improves existing ones - Commits each change - Runs coverage check - Keeps if coverage increases, reverts if it decreases or tests fail - Repeats until target reached or interrupted ### 2. Eliminate TypeScript `any` Types ```text You: $codex-autoresearch Get rid of all the any types in my TypeScript code Codex: Found 47 `any` occurrences in src/**/*.ts Metric: `any` count (current: 47), direction: lower Verify: grep -r "any" src | wc -l Guard: npm run type-check (tsc --noEmit) Run mode? You: Background, run overnight. Codex: Starting background run — baseline: 47. Iterating. ``` **Background mode:** - Runs in detached process - Logs to `autoresearch-results/results.tsv` - Can be stopped/resumed - Check progress: `tail -f autoresearch-results/results.tsv` ### 3. Fix Failing Tests ```text You: $codex-autoresearch Fix the 12 failing tests Codex: Detected 12 test failures via pytest Strategy: Repair one failure at a time Verify: pytest (exit code 0 = success) Mode? You: Foreground. Codex: Iteration 1: test_auth_token_expiry Root cause: Clock skew in mock time... ``` ### 4. Optimize Performance ```text You: $codex-autoresearch Reduce API latency below 100ms Codex: Current p95 latency: 247ms (from load test logs) Target: <100ms Verify: python benchmark.py --p95 Guard: pytest integration tests Mode? You: Foreground. ``` ### 5. Security Audit ```text You: $codex-autoresearch Is this code secure? Codex: Mode: security audit Framework: STRIDE + OWASP Top 10 Scope: src/**/*.py (detected Flask app) Findings will be logged with code evidence. Run mode? You: Foreground. Codex: Finding 1/7: SQL Injection risk in user_search() Evidence: Line 42, raw string interpolation... ``` ## Modes Reference Codex infers the mode from your natural language, but understanding them helps you craft better prompts. | Mode | Trigger Patterns | What It Does | |------|-----------------|--------------| | **loop** | "improve X", "reduce Y", "optimize Z" | Iterative improvement until target or interrupt | | **plan** | "analyze", "what should I improve?", "suggest metrics" | Scans repo, proposes goals and metrics | | **debug** | "why is X happening?", "diagnose", "root cause" | Hypothesis-driven debugging with falsifiable tests | | **fix** | "fix the N failing tests", "repair", "make tests pass" | Sequential repair of known failures | | **security** | "is this secure?", "audit", "STRIDE", "OWASP" | Security analysis with structured findings | | **ship** | "ship it", "ready to release?", "pre-deploy check" | Release readiness verification | | **exec** | (CI/CD usage, see below) | Non-interactive automation mode | ## Configuration **You typically don't need to write config.** Codex infers from your repo and goal. But if you want explicit control, Codex will show you the inferred config and let you adjust before starting: ```text Codex: Inferred configuration: goal: "eliminate any types" scope: "src/**/*.ts" metric: any_count current_value: 47 direction: lower verify_cmd: "grep -r 'any' src | wc -l" guard_cmd: "npm run type-check" Adjust anything? You: Change scope to include test files too. Codex: Updated scope: "{src,test}/**/*.ts" Current value: 63 (including tests) Proceed? ``` ### Explicit Config (Advanced) For CI/CD or scripted use, you can provide a JSON config: ```json { "goal": "Reduce bundle size", "metric": "bundle_kb", "current_value": 487, "target_value": 300, "direction": "lower", "verify_cmd": "npm run build && du -k dist/bundle.js | cut -f1", "guard_cmd": "npm test", "scope": "src/**/*.{ts,tsx}", "max_iterations": 50 } ``` ```bash # Save to file, then: codex exec -f autoresearch_config.json ``` ## Results and State Files All runs create an `autoresearch-results/` directory in your workspace root: ``` autoresearch-results/ ├── results.tsv # Full experiment log (audit trail) ├── state.json # Resume state (last consistent checkpoint) ├── lessons.json # Cross-run learning (what worked/failed) └── sessions/ └── 2026-05-16_14-23-01/ ├── experiment_1_keep.diff ├── experiment_2_discard.diff └── ... ``` ### Reading the Results Log ```bash # View all experiments cat autoresearch-results/results.tsv # Watch live (during background run) tail -f autoresearch-results/results.tsv # Filter successful improvements grep "keep" autoresearch-results/results.tsv ``` **Example log:** ``` iteration commit metric delta status description 0 a1b2c3d 47 0 baseline initial any count 1 b2c3d4e 41 -6 keep replace any in auth module 2 - 49 +8 discard generic wrapper introduced new anys 3 d4e5f6g 38 -3 keep type-narrow API response handlers 4 e5f6g7h 38 0 discard refactor had no effect 5 f6g7h8i 35 -3 keep infer types from JSON schema ``` ## Escalation Strategy (When Stuck) The loop doesn't blindly retry. It escalates: | Trigger | Action | |---------|--------| | 3 consecutive failures | **REFINE** — Adjust within current strategy | | 5 consecutive failures | **PIVOT** — Try fundamentally different approach | | 2 PIVOTs without progress | **Web search** — Look for external solutions | | 3 PIVOTs without progress | **STOP** — Report that human input needed | **One success resets all counters.** Example: ```text Iteration 12: discard (3rd consecutive failure) → REFINE: Try smaller type changes, one file at a time Iteration 17: discard (5th consecutive failure) → PIVOT: Switch from manual typing to codegen from OpenAPI spec Iteration 23: discard (2nd PIVOT without progress) → Web search: "TypeScript eliminate any types best practices" Found: Use strictNullChecks + noImplicitAny... Iteration 25: keep → Counters reset. Back to normal iteration. ``` ## Stopping and Resuming ### Stop a Foreground Run ```text You: (interrupt Codex) Codex: Autoresearch interrupted. State saved to autoresearch-results/state.json Resume with: $codex-autoresearch ``` ### Stop a Background Run ```text You: $codex-autoresearch Codex: Background run detected (PID 12345). Status: 15 iterations, metric improved 47→35 You: Stop it. Codex: Stopping background run... Final state saved. Resume anytime with $codex-autoresearch. ``` ### Resume ```bash $codex-autoresearch ``` Codex automatically detects `state.json` and offers to resume: ```text Codex: Found interrupted session: Goal: eliminate any types Progress: 47→35 (12 iterations) Last commit: f6g7h8i Resume from iteration 13? You: Yes, continue. ``` ## Dual-Gate Verification Every iteration runs **two checks**: 1. **Verify** — Did the metric improve? 2. **Guard** — Did anything break? ```python # Pseudocode of each iteration git checkout -b experiment_N modify_code() git commit -m "experiment N: {hypothesis}" verify_result = run(verify_cmd) guard_result = run(guard_cmd) if verify_result.improved and guard_result.passed: git merge experiment_N log("keep") else: git reset --hard HEAD~1 log("discard") ``` **Example:** - **Verify:** `pytest --cov` (did coverage increase?) - **Guard:** `pytest` (did all tests still pass?) A change that increases coverage but breaks tests is **discarded**. ## CI/CD Mode (exec) For automation pipelines, use `exec` mode: ```bash # Non-interactive, JSON output, exit codes codex exec -f config.json --max-iterations 20 --timeout 3600 ``` **Exit codes:** - `0` — Target reached - `1` — Max iterations reached without target - `2` — Error or guard failure **Example config.json:** ```json { "goal": "Reduce lint warnings to zero", "metric": "lint_warnings", "current_value": 34, "target_value": 0, "direction": "lower", "verify_cmd": "npm run lint -- --format json | jq '.length'", "guard_cmd": "npm test", "max_iterations": 30 } ``` **Output (JSON):** ```json { "status": "success", "iterations": 18, "initial_value": 34, "final_value": 0, "improvements": 12, "discards": 6, "elapsed_seconds": 1847 } ``` ## Advanced Features ### Parallel Experiments (Git Worktrees) Test up to 3 hypotheses simultaneously: ```text You: $codex-autoresearch Try multiple approaches to reduce latency Codex: Parallel mode enabled (3 worktrees). Hypothesis A: Database query optimization Hypothesis B: Response caching Hypothesis C: Async I/O refactor Testing in parallel... Results: A: -23ms (keep) B: -67ms (keep) ← Best C: +12ms (discard) Merging B, continuing from new baseline. ``` ### Cross-Run Learning The `lessons.json` file accumulates knowledge: ```json { "successful_patterns": [ { "goal": "reduce any types", "approach": "infer from JSON schema", "success_rate": 0.83, "avg_improvement": 4.2 } ], "failed_patterns": [ { "goal": "reduce any types", "approach": "generic type wrappers", "failure_rate": 0.91, "reason": "introduced more anys downstream" } ] } ``` Future runs bias toward proven approaches and away from known failures. ### Session Hooks (Auto-Persistence) Hooks keep Codex on track across session boundaries: ```bash # Auto-installed with skill ~/.config/codex/hooks/post-session.sh ``` On every Codex session end: 1. Saves current state to `state.json` 2. Commits results log to git (if repo) 3. Backs up lessons learned **Manual hook setup (if needed):** ```bash chmod +x ~/.config/codex/skills/codex-autoresearch/hooks/post-session.sh ln -s ~/.config/codex/skills/codex-autoresearch/hooks/post-session.sh \ ~/.config/codex/hooks/post-session.sh ``` ## Real-World Examples ### Python: Improve Type Coverage ```python # Before autoresearch def process_data(data): # No types result = [] for item in data: result.append(item['value'] * 2) return result # After 8 iterations from typing import List, Dict, Any def process_data(data: List[Dict[str, Any]]) -> List[float]: result: List[float] = [] for item in data: