harness-synthesizer

name	harness-synthesizer
description	Synthesize code harnesses for agent action validation — AutoHarness-inspired verifier/filter/policy generation
scope	core
version	1.0.0
user-invocable	true
argument-hint	[--mode verifier|filter|policy] [--agent <name>] [--dry-run]
effort	high

Harness Synthesizer Skill

Purpose

Synthesize executable validation harnesses for agent tool calls, inspired by AutoHarness (Google DeepMind, arxiv 2603.03329). Generates code-level verifiers that check action validity before or after execution, reducing agent errors through structured constraint enforcement.

Default mode is advisory (verifier). Hard enforcement requires explicit --hard-enforce opt-in per R021.

Three Modes

Mode	Flag	Behavior	Enforcement
verifier	default	Post-hoc check: validates tool call results after execution	Advisory only
filter	--mode filter	Pre-execution check: blocks invalid tool calls	Opt-in, requires --hard-enforce
policy	--mode policy	Suggests the best valid action from available options	Advisory only

Verifier Mode (Default)

Generates a YAML harness that describes post-execution checks for each tool the agent uses. Checks are emitted as advisory warnings — they do not block execution.

# Example verifier harness output
harness:
  agent: lang-golang-expert
  mode: verifier
  rules:
    - tool: Write
      checks:
        - field: file_path
          pattern: ".*\\.go$"
          on_fail: warn  # advisory
        - field: content
          must_not_contain: "TODO:"
          on_fail: warn
    - tool: Bash
      checks:
        - command_pattern: "^(go build|go test|go fmt|go vet)"
          on_fail: warn

Filter Mode (Opt-in)

Generates pre-execution filter rules. Requires --hard-enforce flag. Used when advisory warnings are insufficient and the risk of invalid actions is high.

# Example filter harness output (--hard-enforce)
harness:
  agent: mgr-gitnerd
  mode: filter
  enforcement: hard
  rules:
    - tool: Bash
      blocks:
        - pattern: "git push --force"
          reason: "Force push to protected branch"
        - pattern: "git reset --hard"
          reason: "Destructive reset without confirmation"

Policy Mode

Generates a policy function that ranks valid actions and suggests the best one. Useful for agents with multiple valid paths to the same goal.

# Example policy harness output
harness:
  agent: qa-engineer
  mode: policy
  policies:
    - scenario: "test file modification"
      preferred_sequence:
        - tool: Read
          reason: "Read before modifying"
        - tool: Edit
          reason: "Edit is safer than Write for existing files"
      avoid:
        - tool: Write
          on_existing_file: true
          reason: "Overwrites without diff"

Workflow

1. Read target agent frontmatter — extract tools, domain, limitations fields
2. Analyze recent tool call patterns — check .claude/outputs/ for prior session logs (if available)

Tool: Writing artifacts under .claude/outputs/

Under mode: "bypassPermissions", direct Write/Edit/Bash on .claude/** paths (including .claude/outputs/sessions/) is permitted (CC v2.1.121+, #1101) — no /tmp wrapping is needed.

Write harness-synthesizer results directly to .claude/outputs/sessions/$(date +%Y-%m-%d)/harness-synthesizer-$(date +%H%M%S).md. Read-only Bash on .claude/outputs/ (e.g., cat, head, wc) is allowed for verification. Catastrophic shell operations (e.g., rm -rf /) remain blocked by independent safety guards. For CC < v2.1.121, see git history for the legacy /tmp/*.sh bypass pattern.

Reference: R006 Sensitive Path Handling, R010 Universal bypassPermissions, #1101.

3. Synthesize validation harness — generate YAML harness matching agent's declared capabilities
4. Refine via evaluator-optimizer loop — iterate harness against edge cases (3 rounds max)
5. Save output — write to .claude/outputs/harnesses/{agent-name}-{mode}.yaml
6. Report — print harness summary and integration instructions

Integration

System	How
action-validator	Harness output feeds into action-validator's code-verified mode
adaptive-harness --learn	Auto-triggers harness-synthesizer for project-specific patterns
evaluator-optimizer	Provides iterative refinement loop (gradient-free optimization)
pipeline-guards	Harness checks usable as pipeline quality gates

Usage Examples

# Generate advisory verifier for lang-golang-expert
/harness-synthesizer --agent lang-golang-expert --mode verifier

# Dry-run: preview harness without saving
/harness-synthesizer --agent mgr-gitnerd --mode filter --dry-run

# Generate hard-enforce filter (explicit opt-in)
/harness-synthesizer --agent mgr-gitnerd --mode filter --hard-enforce

# Generate policy harness
/harness-synthesizer --agent qa-engineer --mode policy

R021 Compliance

• Default verifier mode: advisory only — never blocks tool execution
• filter mode without --hard-enforce: advisory only — emits warnings
• filter --hard-enforce: opt-in hard enforcement — requires explicit user flag
• All harness output is saved to .claude/outputs/harnesses/ (git-untracked)

Output Format

Harnesses are saved as YAML at .claude/outputs/harnesses/{agent-name}-{mode}.yaml. Each harness includes:

harness:
  agent: {agent-name}
  mode: verifier | filter | policy
  version: 1.0.0
  generated: {ISO-8601 timestamp}
  enforcement: advisory | hard  # hard only with --hard-enforce
  rules: [...]

2-Stage Isolation Example (v0.108.0 via #986)

Source: #986 (Deep Insight Part 3 후속, guides/harness-engineering/ Section 3 확장)

harness 생성 시 격리 수준을 단계적으로 적용하는 2-Stage Isolation 패턴 구체 예시.

Stage 1 — Base64 Encoding (input isolation)

에이전트 작업 입력 (특히 외부 소스에서 온 데이터)을 직접 shell/Python 컨텍스트에 주입하지 않고 Base64 인코딩한 후 runtime에서 디코드:

import base64

# Before (unsafe): 직접 삽입
# subprocess_run(["python", "-c", f"process('{user_input}')"])

# After (Stage 1): Base64 격리
encoded = base64.b64encode(user_input.encode()).decode()
subprocess_run([
    "python", "-c",
    f"import base64; process(base64.b64decode('{encoded}').decode())"
])

방어 대상: shell metacharacter 주입, quote escape 우회, 다중라인 페이로드.

Stage 2 — Subprocess Isolation (execution isolation)

Stage 1 인코딩 후에도, 실제 실행은 격리된 subprocess에서 수행. 메인 에이전트 context에서 직접 eval/exec 금지:

# Before (unsafe): 메인 프로세스에서 직접 실행
# exec(decoded_code)  # 에이전트 메모리 오염 가능

# After (Stage 2): subprocess + 리소스 제한
result = subprocess_run(
    ["python", "-c", safe_runner_script, encoded],
    timeout=30,
    capture_output=True,
    env={"PATH": "/usr/bin"},  # PATH 제한
)

방어 대상: 무한 루프(timeout), 파일 시스템 접근(env 제한), 부모 에이전트 상태 오염.

Verifier/Filter/Policy Generation Before/After

harness-synthesizer가 생성하는 verifier/filter/policy에 2-Stage Isolation을 기본 활성화:

Before (v0.107.0 이전):

• verifier는 문자열 매칭만 수행
• filter는 정규식 기반 단순 차단
• policy는 허용/거부 리스트

After (v0.108.0, 2-Stage 기본 적용):

• verifier: Base64 인코딩 후 subprocess AST 파싱 → 구조 검증
• filter: Stage 1 디코드 후 AST 레벨 필터 (semantic, not lexical)
• policy: subprocess env 제한 + capability token (capability-based security)

Action Items

• harness-synthesizer 호출자는 기본으로 2-Stage 활성화 가정
• verifier/filter/policy 커스텀 시 Stage 1/2 모두 유지 (한 단계 건너뛰지 말 것)

Cross-references

• #986 (source)
• guides/harness-engineering/README.md Section 3
• #976 (Deep Insight Part 3 내재화 — 이 섹션의 기반)

Related Guide

• guides/harness-engineering/ — 하네스 엔지니어링 통합 가이드 (3-Layer Hierarchy + Context Engineering + Behavior/Isolation 원칙)