agent-learnings-portability

name	agent-learnings-portability
version	1.1
description	Git-track AI agent learnings so they survive machine loss — memory snapshots, corrections, patterns, insights across Codex, Hermes, Codex, and Gemini.
tags	["harness","git","portability","memory","agent-state","gitignore"]
trigger	When auditing agent state portability, setting up a new machine, or investigating why gitignore exceptions aren't working
effort	medium

Agent Learnings Portability

Problem

AI agents (Codex, Hermes, Codex, Gemini) accumulate valuable learnings in machine-local directories (~/.Codex/, ~/.hermes/, ~/.codex/, ~/.gemini/). Without explicit git tracking, these are lost on machine failure.

Architecture: Three Tiers

1. Git-committed in .Codex/memory/ (survives git clone, PRIMARY): Memory facts, conventions, user context — travels with the repo
2. Rsync backup (survives single-machine failure): Large files — session transcripts, SQLite databases, full Codex project memory dirs
3. Regenerable (no backup needed): Caches, plugins, debug logs

The .Codex/memory/ Pattern (issue #1886)

The primary mechanism for cross-machine context parity is writing canonical memory facts directly into .Codex/memory/ inside the repo. This directory is already git-tracked and read by Codex at session start on every machine.

How it works:

1. Hermes memory (~/.hermes/memories/MEMORY.md + USER.md) is the authoritative source
2. Bridge script (scripts/memory/bridge-hermes-Codex.sh) reads Hermes memory + Codex auto-memory
3. Writes unified entries into .Codex/memory/agents.md and context.md
4. git commit + push → every machine that pulls gets the same context
5. Windows (no Hermes): git pull → done. No manual copying, no tarballs, no installers.

Key files:

• .Codex/memory/context.md — Machine conventions, paths, Python commands, workspace layout
• .Codex/memory/agents.md — User profile, AI subscriptions, workflow rules, GSD facts
• .Codex/memory/Codex-auto-memory.md — Snapshot of Codex's auto-generated MEMORY.md
• scripts/memory/bridge-hermes-Codex.sh — The bridge script, run with --commit flag

Why not tarballs or export/import scripts? One-time snapshots go stale immediately. Git gives version history, diffs, rollback, and automatic updates on every git pull. The bridge should run daily via cron, not manually.

Return enrichment flow: Non-Hermes machines (Windows) enrich KNOWLEDGE.md or topic files → git commit/push → Linux pulls → bridge script picks up on next run via Codex auto-memory snapshot capture.

This pattern REPLACES the older config/agents/ snapshot approach for memory (keep config/agents/ for large state files like session exports, but use .Codex/memory/ for human-readable context).

What to Git-Track (by agent)

Hermes (~/.hermes/)

• memories/MEMORY.md + USER.md → snapshot to config/agents/hermes/memories/

Codex (~/.Codex/)

• projects/<encoded-path>/memory/*.md → snapshot to config/agents/Codex/memory-snapshots/
  • Especially feedback_*.md (user corrections) and project_*.md (context)
• .Codex/state/ directories (in-repo): corrections/, patterns/, reflect-history/, cc-insights/, trends/, candidates/, session-signals/, skill-eval-results/
• .Codex/state/ files: learned-patterns.json, skill-scores.yaml, cc-user-insights.yaml, hermes-insights.yaml, cross-agent-memory.yaml, drift-summary.yaml, portfolio-signals.yaml

Codex (~/.codex/)

• rules/default.rules (learned permissions — CRITICAL)
• history.jsonl, session_index.jsonl → snapshot to config/agents/codex/state-snapshots/

Gemini (~/.gemini/)

• state.json, projects.json → snapshot to config/agents/gemini/state-snapshots/

CRITICAL PITFALL: Gitignore Directory vs Glob

This is the #1 gotcha that makes gitignore exceptions silently fail.

The Bug Pattern

# THIS IS BROKEN — exceptions below are DEAD CODE
.Codex/state/        # <-- trailing slash = ignores the DIRECTORY
!.Codex/state/corrections/   # <-- NEVER WORKS

When git ignores a directory (trailing /), it never looks inside it, so child exceptions cannot un-ignore anything.

The Fix (three layers required)

# Layer 1: Top-level ignore with glob (not directory)
.Codex/*

# Layer 2: Un-ignore the parent directory itself
!.Codex/state/

# Layer 3: Re-ignore contents, THEN add specific exceptions
.Codex/state/*
!.Codex/state/corrections/
!.Codex/state/patterns/
!.Codex/state/learned-patterns.json

Verification

# Check if a file is ignored (should show NO output for tracked files)
git check-ignore -v .Codex/state/corrections/foo.jsonl

# If it shows a rule, the exception is NOT working
# The -v flag shows WHICH rule is blocking

Debugging Workflow (when exceptions don't work)

# Step 1: Check which rule is blocking
git check-ignore -v .Codex/state/corrections/foo.jsonl

# Output shows the EXACT line: ".gitignore:127:.Codex/*"
# Step 2: Check if there are MULTIPLE blocking rules
git check-ignore -v .Codex/state/corrections/ .Codex/state/learned-patterns.json

# Step 3: After fixing, verify files CAN be staged
git add .Codex/state/corrections/ 2>&1
# If you see "ignored by .gitignore" there's still a blocking rule

Common Failure Modes

1. Parent dir ignored with / instead of /* — children can never be excepted
2. Multiple ignore rules at different levels — e.g., .Codex/* at line 127 AND .Codex/state/ at line 157. You fix one but the other still blocks. Always run git check-ignore -v AFTER each fix to verify
3. Exception added but files never git add-ed — the gitignore exception alone doesn't track files
4. Cron uses inline command instead of script — after updating a backup script, verify the crontab actually calls the script (not an old inline rsync)

Nightly Pipeline Integration

Script: scripts/cron/commit-learning-artifacts.sh

Steps:

1. Snapshot agent memories from home dirs to config/agents/*/
2. git add all state directories and snapshot directories
3. Run legal-sanity-scan.sh --diff-only (MANDATORY — corrections and session data can contain client names)
4. Commit and push if changed

Wire into: comprehensive-learning-nightly.sh as final step.

Session-Signal Redaction

session-signals/*.jsonl files contain a last_assistant_message field with full LLM response text. This text often references client project names (from doc-intelligence classification outputs). The field must be redacted before git-add.

Script: scripts/cron/redact-session-signals.sh

• Scans all .Codex/state/session-signals/*.jsonl
• Replaces last_assistant_message with [REDACTED] using Python JSON parsing
• Preserves all metadata fields (session_id, hook_event_name, permission_mode, etc.)
• Must run BEFORE git add in the nightly pipeline

Codex Session Export

Codex sessions live at ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl in a different format than Codex/Hermes.

Key Codex JSONL types:

• session_meta — session ID, cwd, model provider
• response_item with payload.type == "function_call" — tool calls
• response_item with payload.type == "function_call_output" — results
• Tool names: exec_command, read_file, write_file, apply_diff, write_stdin, list_directory

Script: scripts/cron/codex-session-export.sh

• Exports to logs/orchestrator/codex/session_YYYYMMDD.jsonl
• Maps Codex tool names to orchestrator conventions (exec_command→Bash, etc.)
• Incremental via .last-export-ts timestamp file

Legal Risk

Session signals, corrections, and memory files can contain client project names. The legal scan MUST gate all commits. If it finds violations, skip the commit rather than pushing sensitive data.

Key mitigation: run redact-session-signals.sh before staging. The session-signals directory is the #1 source of legal violations (522 hits in first attempt without redaction).

Backup Coverage (Tier 2)

scripts/cron/memory-backup.sh rsyncs to ace-linux-2 daily:

• Codex project memory (~/.Codex/projects/)
• Hermes memories + sessions (~/.hermes/memories/, ~/.hermes/sessions/)
• Codex sessions + rules (~/.codex/sessions/, ~/.codex/rules/)
• Gemini sessions (~/.gemini/tmp/)

Use ssh -o ConnectTimeout=10 -o BatchMode=yes for non-interactive cron. Non-critical agents use || true to not abort.

Restore on New Machine

Script: scripts/_core/sync-agent-configs.sh (restore section)

Behavior: only copies if target files are MISSING (safe on existing machines). Use --force to overwrite.

• Hermes: copies .snapshot files → ~/.hermes/memories/ (strips .snapshot suffix)
• Codex: copies config/agents/Codex/memory-snapshots/*.md → encoded project path, additive (skips existing)
• Codex: copies default.rules, history.jsonl, session_index.jsonl
• Gemini: copies state.json, projects.json

Codex project path encoding: /mnt/local-analysis/workspace-hub → -mnt-local-analysis-workspace-hub