autoconference-analyze

name	autoconference:analyze
description	Post-conference analysis — compares researcher trajectories, identifies failure modes, extracts transferable insights, and produces an insight taxonomy. TRIGGER when: user wants to analyze conference results, understand what happened, extract learnings from a completed conference. DO NOT TRIGGER when: user wants to run a conference (use autoconference) or ship results (use ship).
allowed-tools	["Agent","Read","Write","Edit","Bash","Glob","Grep","WebSearch"]

autoconference:analyze — Post-Conference Insight Analysis

Analyze a completed autoconference to extract insights, patterns, and transferable learnings. This skill reads all conference artifacts and produces a structured analysis-report.md.

Chaining position: autoconference → analyze → ship

---

Pre-Analysis Setup

Before running analysis, verify the conference directory exists and is complete.

1. Ask the user: "Which conference directory should I analyze?" (if not already specified)
2. Confirm the directory contains at minimum: conference.md and conference_results.tsv
3. If final_report.md does not exist, warn: "No final_report.md found — analysis will work from raw researcher results only. Continue?"
4. Set CONF_DIR to the conference directory path for all subsequent reads.

---

Step 1: Load Conference Data

Read all available conference artifacts from CONF_DIR. Do not skip files — the more context, the better the analysis.

Required files (fail if missing):

• conference.md — Extract: goal, metric, baseline value, researcher count, max_rounds, strategy assignments
• conference_results.tsv — The aggregated results table

Optional files (read if present, skip gracefully if absent):

• researcher_{ID}_results.tsv for each researcher ID found in conference.md — Individual iteration histories
• poster_session_round_*.md — Knowledge sharing transcripts from each round's symposium phase
• peer_review_round_*.md — Adversarial review transcripts from each round's review phase
• synthesis.md — Intermediate synthesis notes if generated mid-conference
• final_report.md — The conference's own synthesis output

Extraction targets while reading:

• From conference.md: goal statement, target metric name and direction (maximize/minimize), baseline metric value, number of researchers, their assigned strategies, whether a Devil's Advocate was assigned
• From conference_results.tsv: columns (researcher_id, round, iteration, metric_value, strategy_description, kept/reverted, notes)
• From poster session files: what knowledge was shared, which researcher shared it, which round
• From peer review files: what claims were challenged, what was overturned, reviewer reasoning

---

Step 2: Trajectory Comparison

For each researcher, compute the following metrics from their researcher_{ID}_results.tsv:

Convergence speed: Number of iterations from start to the iteration where the researcher first achieved their personal best metric. Lower is faster.

Exploration breadth: Count of distinct strategy categories tried. Strategies are "distinct" if they describe fundamentally different approaches (e.g., "regularization tuning" vs. "architecture change" are distinct; "L1 regularization" vs. "L2 regularization" are the same category). Use your judgment to cluster.

Success rate: (number of iterations marked 'kept') / (total iterations). Express as a percentage.

Final metric vs baseline improvement: (final_best_metric - baseline_metric) / baseline_metric * 100%. Label as improvement or regression depending on metric direction.

Notable finding: One sentence describing the most significant discovery or pattern for this researcher.

Produce the following comparison table in the report:

Researcher	Assigned Strategy	Best Metric	Convergence Speed (iters)	Exploration Breadth	Success Rate	Final Improvement vs Baseline	Notable Finding
R1	...	...	...	...	...	...	...
R2	...	...	...	...	...	...	...
...	...	...	...	...	...	...	...

After the table, write 2-3 sentences summarizing the key trajectory differences. Who converged fastest? Who explored most broadly? Was there a trade-off between speed and final metric?

---

Step 3: Failure Mode Analysis

Identify patterns of failure across all researchers. A "failure" is any iteration marked reverted or any claim marked challenged/overturned in peer review.

Substep 3a — Reverted strategy clustering: Go through all researcher TSV files and collect every row where kept = reverted. Group these by strategy type (same clustering logic as Step 2). For each cluster, record:

• Strategy type name
• How many times it was tried across all researchers
• How many researchers tried it
• Whether it ever worked for anyone (any kept row in the same cluster)

Present as a table:

Strategy Type	Times Tried	By N Researchers	Ever Worked?	Verdict
...	...	...	...	...

Verdict should be one of: dead end (never worked), researcher-specific (worked for some but not others), timing-dependent (worked in later rounds but not early), conditional (worked only with specific prerequisites).

Substep 3b — Peer review overturns: From peer_review_round_*.md files, list every claim that was challenged or overturned. For each:

• The claim made
• The round it was challenged in
• The reason given for the challenge
• Whether the researcher recovered (did they produce a better result after the challenge?)

Substep 3c — Systematic failure patterns: Look for failures that afflicted multiple researchers in the same round. If 3+ researchers tried the same type of strategy and it failed in round N, that is a systematic failure. Flag these explicitly: "Systematic failure in Round N: [description]. Possible cause: [hypothesis]."

---

Step 4: Cross-Researcher Insight Extraction

Analyze knowledge transfer effectiveness using the poster_session_round_*.md files.

Substep 4a — Knowledge transfer audit: For each piece of knowledge shared in a poster session:

• Who shared it
• What round it was shared in
• Which researchers were present (all, unless the file indicates otherwise)
• Whether any researcher subsequently adopted that technique (check their TSV rows after that round)
• The metric impact of adoption (compare metric in the round before adoption vs. 2 rounds after)

Substep 4b — Transfer effectiveness table:

Shared Knowledge Item	Source Researcher	Round Shared	Adopters	Non-Adopters	Avg Metric Gain for Adopters
...	...	...	...	...	...

Substep 4c — Breakthrough propagation: Identify whether any single researcher's breakthrough had outsized influence. A "breakthrough" is any iteration that produced a metric improvement >2x the average per-iteration improvement for that researcher. If a breakthrough was shared and caused measurable improvements in others, record:

• "Researcher R{ID}'s discovery of [technique] in Round N propagated to R{ID2} and R{ID3}, producing combined improvements of X%."

If no poster session files exist, write: "No poster session transcripts available — transfer analysis skipped."

---

Step 5: Insight Taxonomy

Produce a structured taxonomy of all actionable insights extracted from the conference. An insight is any finding that could be applied to a future similar problem.

Assign each insight a type from this controlled vocabulary:

• technique — A specific method or approach that improved the metric
• anti-pattern — A specific method or approach that consistently failed
• meta-strategy — An insight about how to run the search process itself (e.g., "explore broadly in early rounds, exploit in later rounds")
• domain-insight — A fact about the problem domain that influenced strategy selection
• tooling — An observation about tools, libraries, or infrastructure

Confidence scoring:

• high — Observed by 2+ researchers, consistent result, no contradictory evidence
• medium — Observed by 1 researcher with clear causal mechanism, or by 2+ researchers with one contradictory instance
• low — Observed once, causation unclear, or contradicted by another researcher

Transferability:

• yes — Applies broadly to similar problems in this domain
• conditional — Applies only under specific conditions (describe them)
• no — Specific to this dataset/problem instance

#	Insight	Type	Source Researcher(s)	Round First Observed	Confidence	Transferable?	Conditions (if conditional)
1	...	...	...	...	...	...	...
2	...	...	...	...	...	...	...
...	...	...	...	...	...	...	...

Aim for completeness — a thin taxonomy is a failed analysis. Extract at least one insight per researcher per round if the data supports it.

---

Step 6: Novelty Assessment

For each insight in the taxonomy (Step 5), score it on two additional dimensions:

Predictability score (was this obvious from the baseline?):

• obvious — Any domain expert would have predicted this from the problem statement
• plausible — An expert might have guessed this, but it wasn't certain
• surprising — This would not have been predicted without running the conference

Multi-agent advantage (did the conference format enable this discovery?):

• yes — This insight required multiple researchers, knowledge transfer, or adversarial review to emerge
• possibly — Could have been found by a single researcher, but conference accelerated it
• no — Any single researcher running enough iterations would have found this

Add these columns to the insight taxonomy table, or present as a supplementary table referencing insight numbers.

After the table, write a "Conference ROI" paragraph: Given the total number of researcher-iterations run, how much insight was produced? Was the conference format justified, or could a single deep autoresearch loop have achieved similar results? Be honest — if the answer is "a single researcher would have been sufficient," say so.

---

Step 7: Write Report

Write {CONF_DIR}/analysis-report.md with the following structure. Do not truncate any section — each section must be fully populated.

# Conference Analysis Report

**Conference Goal:** [from conference.md]
**Baseline Metric:** [value and metric name]
**Best Achieved Metric:** [value, by which researcher, in which round]
**Total Researcher-Iterations:** [sum across all researchers]
**Analysis Generated:** [current date]

---

## 1. Trajectory Comparison
[Full table and summary from Step 2]

## 2. Failure Mode Analysis

### 2a. Reverted Strategy Clusters
[Table from Step 3a]

### 2b. Peer Review Overturns
[List from Step 3b]

### 2c. Systematic Failure Patterns
[Findings from Step 3c, or "None detected."]

## 3. Cross-Researcher Insight Extraction

### 3a. Knowledge Transfer Audit
[Table from Step 4b]

### 3b. Breakthrough Propagation
[Findings from Step 4c]

## 4. Insight Taxonomy
[Full table from Step 5]

## 5. Novelty Assessment
[Extended table from Step 6 + Conference ROI paragraph]

## 6. Recommended Next Steps

[Write 3-5 concrete, actionable recommendations based on the analysis. Each recommendation should cite at least one specific insight from the taxonomy. Format as a numbered list with one sentence of justification per item.]

After writing, confirm: "Analysis complete. Report written to {CONF_DIR}/analysis-report.md. [N] insights extracted, [M] failure modes identified. Top recommendation: [first item from §6]."

---

Error Handling

Missing researcher TSV files: Proceed with conference_results.tsv only. Note in the report which researchers had no individual TSV available and flag that trajectory analysis for those researchers is approximate.

Truncated conference (stopped before max_rounds): Note the stopping round prominently at the top of the report. Distinguish between: user-interrupted, convergence-triggered, or error-stopped. Adjust trajectory analysis accordingly.

No poster session files: Skip Step 4 and note the absence. Knowledge transfer analysis requires these files.

Contradictory data (e.g., a strategy marked both kept and reverted in different rows): Flag it explicitly in the relevant section and use the later-round entry as the authoritative record.