research-wiki

name

research-wiki

description

Maintain a Karpathy-style LLM-wiki (https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) over the accumulating progress_*.md files in /home/dgu_wj92/ysl/StealthyIPIAttack/progress. Supports three operations — ingest (fold a new progress file into entity/concept pages), query (answer a natural-language research question by reading relevant wiki pages and citing sources), lint (detect stale, contradictory, or orphan pages). Use when the user says "wiki ingest", "wiki query", "wiki lint", "지금까지 MELON 뚫은 공격 정리해줘", "wiki에 넣어줘", or similar.

research-wiki

Persistent, compounding knowledge base over AgentDojo attack-research progress reports. Inspired by Karpathy's LLM-wiki pattern: raw sources stay immutable, the wiki is LLM-maintained with cross-references, and three operations (ingest / query / lint) keep it coherent.

Paths

REPO=/home/dgu_wj92/ysl/StealthyIPIAttack            # git repo root
PROGRESS_ROOT=/home/dgu_wj92/ysl/StealthyIPIAttack/progress  # progress_<researcher>/progress_YYYYMMDD_HHMM.md
WIKI=/home/dgu_wj92/ysl/StealthyIPIAttack/wiki       # the wiki (git-tracked)
LIVE=/home/dgu_wj92/ysl/AgenticRAG/Experiment1/AgentDojo  # live workspace (read-only from here)

Wiki layout (dual-wiki: integrated + per-researcher)

wiki/
├── index.md          # integrated catalog: attacks / defenses / concepts / findings
├── log.md            # integrated ingest/query/lint log
├── attacks/<name>.md     # shared attack entity pages (person-agnostic)
├── defenses/<name>.md    # shared defense entity pages
├── concepts/<name>.md    # shared concept pages
├── findings/q_<slug>.md  # optional cross-researcher findings
│
├── ys/               # sub-wiki for researcher ys
│   ├── index.md      #   ys's contributions + links to shared entities
│   ├── timeline.md   #   ys's time-ordered events
│   └── log.md        #   ys sub-wiki operation log
├── yj/               # sub-wiki for researcher yj (same layout)
└── wj/               # sub-wiki for professor wj (feedback/reviews)

Rationale: shared research assets (attacks, defenses, concepts) are person-agnostic and live at the top level. Each researcher's personal view (what they contributed, their timeline) lives in wiki/<researcher>/. Ingest writes to BOTH when applicable.

Detecting the researcher from a progress file path

progress/<researcher>/progress_YYYYMMDD_HHMM.md → researcher = <researcher> (ys/yj/wj).

If the path is ambiguous (e.g. legacy file directly in progress/ root), ask the user which researcher to attribute.

Entity page template (attacks/defenses/concepts)

---
name: <entity-name>
type: attack | defense | concept
status: active | deprecated | superseded
last_updated: YYYY-MM-DD
---

# <Entity display name>

## Summary
<one paragraph, what this is, why it matters>

## Latest Numbers
<only for attacks/defenses — markdown table with model, defense/attack, progress, ASR, Both, Util, source>

## Timeline
- YYYY-MM-DD HH:MM · <short note> · source: `progress_YYYYMMDD_HHMM.md`
- ...

## Cross-references
- [related entity](../<dir>/<name>.md)
- ...

## Design / Mechanism
<only for attacks — payload structure, trigger vocab, etc.>
<only for defenses — algorithm summary, known holes>

Operations

1. Ingest

Input: path to a progress_YYYYMMDD_HHMM.md file. If omitted, default to the newest progress file under $PROGRESS_ROOT/<researcher>/progress_*.md (across all researchers).

Researcher detection: extract <researcher> from the path (progress/<researcher>/...). If path is at the old flat progress/ location, ask which researcher to attribute.

Steps:

1. Read the progress file.
2. Parse its "Current Status (live)" table — each row is (model, attack, defense, progress, ASR, Both, Util, state). Use the progress file's timestamp as the row's datestamp.
3. For each (attack, defense) tuple — update the integrated shared entity pages (person-agnostic, shared across researchers):
   • Upsert wiki/attacks/<attack>.md:
     • Create if missing (use entity template).
     • Append a Timeline bullet with datestamp + one-line result + source cite (citing progress/<researcher>/progress_*.md).
     • Replace the Latest Numbers table row for this (attack, defense) with the newest values.
     • Bump last_updated in frontmatter.
   • Same for wiki/defenses/<defense>.md.
4. Parse "Done" items. If a Done item introduces a NEW attack, defense, or concept (e.g. "신규 공격 trigger_fake 구현 완료"), create the shared entity page if missing.
5. Parse "Known incidents" items — if an incident describes a conceptual finding, create or update the relevant concept page in wiki/concepts/.
6. Update the integrated wiki/index.md (shared catalog, alphabetical within each section).
7. Update the researcher's sub-wiki at wiki/<researcher>/:
   • wiki/<researcher>/timeline.md: append a dated section summarizing what the progress file reported (in the researcher's voice).
   • wiki/<researcher>/index.md: ensure all NEW attacks/concepts introduced in this progress file are linked under "Contributions".
   • wiki/<researcher>/log.md: append - YYYY-MM-DD HH:MM · ingest progress_YYYYMMDD_HHMM.
8. Append to integrated wiki/log.md:

   - YYYY-MM-DD HH:MM · ingest <researcher>/progress_YYYYMMDD_HHMM.md — touched N attacks, M defenses, K concepts
   

9. Commit: git add wiki/ && git commit -m "wiki: ingest <researcher>/progress_YYYYMMDD_HHMM". Do NOT push.

Never modify the progress_*.md file itself. It is an immutable source.

2. Query

Input: a natural-language question (Korean or English).

Steps:

1. Read wiki/index.md to identify candidate entity pages.
2. Read those pages. If pages reference a specific progress file that could add context, read that file too.
3. Synthesize an answer. Cite inline in this format: [attack:trigger] or [progress:20260423_1545]. Every numeric claim must carry a citation.
4. If the answer is substantive (≥ ~100 words of synthesis, likely to be asked again), write it to wiki/findings/q_<slug>.md with:

   ---
   question: <the original question>
   asked_at: YYYY-MM-DD HH:MM
   ---
   
   # <question restated>
   
   <answer body with inline citations>
   
   ## Sources
   - [attack:trigger](../attacks/trigger.md)
   - [progress:20260423_1545](../../progress/progress_20260423_1545.md)
   

   Keep the slug short, lowercase, hyphenated (e.g. q_which-attacks-bypass-melon.md).
5. Append to wiki/log.md:

   - YYYY-MM-DD HH:MM · query "<verbatim question, truncate to 80 chars>" → findings/q_<slug>.md  (or: no-finding if trivial)
   

6. If a finding was written, commit: git add wiki/ && git commit -m "wiki: query <slug>".

3. Lint

No input. Runs checks and emits a report.

Checks:

1. Missing ingests — any progress_YYYYMMDD_HHMM.md not listed in wiki/log.md under an ingest line.
2. Contradictions — same (attack, defense) Latest-Numbers row appears in multiple places or timeline bullets disagree on a number with no explanatory note.
3. Stale pages — entity page last_updated older than 7 days AND referenced in a progress file newer than that (i.e. page should have been updated on ingest but wasn't).
4. Orphans — entity page not linked from wiki/index.md.
5. Dangling citations — page cites progress_*.md or a sibling entity that no longer exists.

Output: print a report. Do NOT auto-fix — user decides. Append to log:

- YYYY-MM-DD HH:MM · lint — P missing-ingests, Q contradictions, R stale, S orphans, T dangling

Invocation convention

• research-wiki ingest [path] — ingest one file. Omit path → newest progress file.
• research-wiki ingest --all — ingest every progress file not yet in log (chronological).
• research-wiki query <question> — answer from wiki.
• research-wiki lint — run health checks.

Constraints

• Immutable sources. Never edit progress_*.md — they are the raw log.
• Cite everything. Any number in an entity page must be followed by a · source: progress_YYYYMMDD_HHMM.md citation.
• No hype. Follow the "no_hype" memory rule — numbers only, no interpretation unless the progress file itself interpreted (and even then, quote with attribution).
• Korean prose, English technical terms. Same as meeting-progress skill.
• Never push. User pushes manually. (Override only when user explicitly asks to push in the same turn.)
• Never auto-trigger. Only run on explicit user request or a meeting-progress skill handoff.
• Contradictions survive. When two progress files disagree on a number (e.g. trigger×ipiguard ASR denom-variant), keep both in Timeline with their dates; Latest Numbers shows the more recent AND notes the earlier discrepancy.
• Entity naming. Use exact attack/defense names as they appear in scripts/run_representative_sweep.py --attack / --defense flags. Examples: trigger, trigger_fake, fake_completion_Nx, chat_inject_qwen3, melon, ipiguard, tool_filter, no_defense.

Handoff from meeting-progress

When meeting-progress finishes and writes a new progress_YYYYMMDD_HHMM.md, it may chain automatically into this skill. Two paths:

1. Auto-chain (same-turn handoff). If the user's original request to meeting-progress contained a chaining keyword (wiki, ingest, 체이닝, ingest까지, 위키), meeting-progress invokes research-wiki ingest <that file> directly after writing the progress file. No second user prompt required.
2. Soft suggestion. Otherwise meeting-progress prints a one-liner suggesting the ingest command; user types approval before this skill runs.

Either way, the ingest target is always the file just written. Never auto-chain for earlier progress files — if the user wants bulk backfill, they must explicitly run research-wiki ingest --all.