メニュー
Map open problems where this repo can advance SOTA. Use when scoping research. Do not use to run the campaign; use night-market-completion-integrity-campaign.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/athola/claude-night-market --skill night-market-research-frontierコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Turn hunches into accepted results: worthiness score, evidence bar, research-to-rules. Use when vetting ideas. Not for QA; use night-market-validation-and-qa.
Generates and self-executes a diff-derived test plan for a PR. Use when validating PR changes before merge. Do not use for code review; use sanctum:pr-review.
Curate the web-capture index. Use when the capture backlog grows, captures sit unprocessed at seedling/pending, or to surface stored research during work.
SKILL.md を表示中
SOC 職業分類に基づく
| name | night-market-research-frontier |
| description | Map open problems where this repo can advance SOTA. Use when scoping research. Do not use to run the campaign; use night-market-completion-integrity-campaign. |
This file lists the open problems where this repository holds assets that the published state of the art does not. SOTA (state of the art) here means the best result shipped or published anywhere, not the best result in this repo. Every entry is a candidate. Nothing below is a claimed capability, and citing this file as evidence that a capability exists is an error.
Read this skill when choosing what research bet to place next, when framing an experiment, or when someone asks "what could this project contribute beyond itself?"
Follow these before starting any problem below.
docs/research/, update the changelog, and remove or re-scope the
entry here. A frontier list that never shrinks is a wish list.| # | Problem | Primary repo asset | Status |
|---|---|---|---|
| 1 | Completion integrity in autonomous loops | egregore gate + herald judge + imbue verifier-integrity | Open, active campaign |
| 2 | Skill-graph governance at scale | forced-eval activation harness + ratchets |
| Open |
| 3 | Collective memory across context resets | ADR-0007 Discussions + memory-palace | Open, partially blocked |
| 4 | Insight-palace bridge under a hook budget | Draft spec v0.1.0 + hook infrastructure | Open, spec drafted |
| 5 | Behavioral contract attestation | ADR-0008 SLSA path + trust workflow | Open |
Autonomous agents self-report "done." The evidence is inlined here
and in the two docs of record it was folded into
(.claude/rules/prefer-invariants-over-fallbacks.md for the
harness-loop findings,
plugins/imbue/skills/proof-of-work/modules/verifier-integrity.md
for the verifier findings): the METR randomized trial
(arXiv 2507.09089)
found experienced developers 19% slower with AI while believing they
were faster, so self-assessment of completion is miscalibrated even
for humans in the loop. On the verifier side, a green check proves
spec-satisfaction, not correctness: the spec can be wrong, or the
check can be hollow (a test that passes no matter what the code does).
An agent that judges its own work optimizes the judge, not the work.
No published harness binds "done" to gates the agent cannot fake.
Three shipped, tested mechanisms that most agent frameworks lack:
completion_integrity: bool = False in
plugins/egregore/scripts/config.py (commit 83281337, default off).
When true, a "fix-required" quality verdict blocks the ship step and
merge is held for human review regardless of auto_merge. The
raw-JSON opt-in path is covered by tests (commit cd903cbf).plugins/herald/hooks/double_shot_latte.py. Deterministic verdict by
default. An optional LLM second shot fires only on the single
ambiguous outcome and is capped at LLM_TIMEOUT_SECONDS = 8 inside
the 10s registered hook budget (commits 3d22f02a, 268cff89).plugins/imbue/skills/proof-of-work/modules/verifier-integrity.md
(commit 29081fda): proves the check was worth passing, distinct from
proving it passed.completion_integrity: true) and again with it off, on the same
work items. Log every quality verdict.A measured false-done rate delta between gated and ungated runs on the same work items exists, with the human adjudication recorded, and the delta survives a second run. If the delta is zero or the gate holds only items a human calls genuinely done, the gate as designed is falsified: record that too. The promotion question (default-off to default-on) is open until this number exists.
This repo carries 197 registered skills (198 SKILL.md files on disk,
find count 2026-07-02) against a finite skill discovery budget of
about 16K characters. Skills past the budget are dropped silently
(docs/quality-gates.md, Follow-on work section). The activation
layer does near-keyword matching, so relevant skills fail to fire
(prototypes/forced-eval/README.md). No one, here or elsewhere, has
published a principled activation-quality metric: a way to say "this
skill library activates the right skill X% of the time, and change Y
moved that number."
prototypes/forced-eval/measure_activation.py (commit 5683e89b).
It runs labeled prompts through claude -p with and without a
forced-eval hook, counts expected Skill() events, tracks
false activations on true-negative cases, and applies a paired
McNemar significance test. The dataset
(prototypes/forced-eval/activation_cases.json) is deliberately
small. The README says to expand it before trusting the rates.scripts/check_skill_graph_drift.py (dangling Skill() refs) and
scripts/check_skill_exit_criteria_drift.py.docs/skill-integration-guide.md.rg -l "Approve with actions" plugins/pensive/skills/*/SKILL.md matches 7 files, 2026-07-02).Expand prototypes/forced-eval/activation_cases.json with labeled
positive and true-negative prompts for the pensive review skills.
Baseline: run the harness dry, then live.
cd prototypes/forced-eval
uv run python measure_activation.py # dry run, spends nothing
uv run python measure_activation.py --live \
--root "$PWD/../../plugins/pensive" --repeats 3
Consolidate pensive:shell-review and pensive:makefile-review
into pensive:unified-review as modules (issue #574 item 1, one PR
per skill, thin command alias stubs kept), then re-run step 2.
A measured activation-lift delta exists for the pensive consolidation: activation rate on the labeled set before versus after, with McNemar significance, plus the discovery-budget character count saved. A result where consolidation saves budget without degrading activation is publishable. A result where activation drops is the falsification and blocks further consolidation. Candidate follow-on, unproven: turn the harness into a CI gate for any skill-count change.
Published agent-memory work centers on single-agent vector stores. Retrieval precision is rarely measured, and nothing binds memory to a team of agents whose contexts reset constantly. The failure mode is documented in this repo's own history: the abstract Stop hook that posts daily [Learning] digests read env vars Claude Code never sets and was a silent no-op for months (fixed in 1.9.14 via the shared stdin-first payload reader). Memory systems fail silently, and nobody notices until the knowledge is needed.
gh discussion subcommand does not exist, so all access is
gh api graphql.plugins/memory-palace/skills/knowledge-intake/modules/discussion-promotion.md
routes reviewed Discussions knowledge into palace storage.plugins/memory-palace/skills/memory-clarity-probe/SKILL.md: dual
anchor questions probing whether a summary preserves task progress
and information gaps across a handoff.plugins/abstract/hooks/post_learnings_stop.py.gh api graphql, and for each
write 1 to 2 queries a future session would plausibly ask.memory-palace:knowledge-locator precision and recall
against that set. Record the numbers in a dated
docs/research/ synthesis.Precision and recall numbers exist for a labeled query set, and one curation change (for example, promoting versus not promoting a batch) produces a predicted, then measured, retrieval delta. Issue #553 unblocks a stronger result (an RL-trained clarity probe), but the retrieval measurement does not wait on it.
Plugin ecosystems either share a runtime registry (tight coupling) or do not exchange data at all. ADR-0001 forbids a shared registry here: plugins detect each other via the filesystem and degrade gracefully. Moving structured findings between two isolated plugins inside a Stop hook's hard latency budget, with graceful failure when the peer plugin is absent, is an unsolved composition problem, and hook-budget overruns are a known repo failure class (herald's LLM timeout once exceeded its registered budget and the harness killed the hook with no verdict at all).
A drafted, unimplemented specification: docs/specification.md
(Insight-Palace Bridge, v0.1.0, Draft, 2026-04-13), with
docs/project-brief.md and docs/implementation-plan.md. Key
verified constraints:
_BUDGET_SECONDS = 8.5 in
plugins/abstract/hooks/post_learnings_stop.py, leaving headroom
inside the 10s hook timeout._HAS_INSIGHT_ENGINE): with memory-palace or the insight engine
missing, the bridge silently does nothing.Caution: the brief, specification, and implementation plan under
docs/ are overwritten per feature cycle. Confirm the spec on disk is
still the insight-palace bridge before building against it.
docs/specification.md and docs/implementation-plan.md end
to end, and confirm the Draft status and version are unchanged._HAS_INSIGHT_ENGINE guard and the remaining-budget check, tests
first (Iron Law applies).sys.meta_path import blocker as the existing hook regression
tests do.The bridge is merged with both tests green, a benchmark artifact shows 10-finding ingestion under 500ms on CI hardware, and the spec's status line moves from Draft. Falsification: if the 500ms budget cannot be met without dropping findings, that is a spec revision, not a reason to remove the budget check.
Supply-chain attestation (SLSA provenance, signed via Sigstore) proves which bytes came from which workflow. It does not prove what the artifact does. ADR-0008 states the gap directly: there is no mechanism to prove that a plugin's behavioral contract holds. A marketplace can today verify a plugin is unmodified and still ship a plugin whose hooks do something other than what its README claims. SLSA is the state of the art for artifacts. Behavior verification has no SOTA to beat, only a vacancy.
.github/workflows/trust-attestation.yml
runs make test on master pushes and produces a signed SLSA
attestation of trust-report.json.leyline:verify-plugin command
(plugins/leyline/commands/verify-plugin.md) checks a plugin's
attestation history.trust-report.json would need to assert for behavior,
not provenance: candidate schema is per-hook contract tests (input
payload, expected verdict/exit) whose pass results are attested.leyline:verify-plugin to compare the attested behavioral
claims against the plugin currently on disk and flag divergence.leyline:verify-plugin distinguishes, in a test, a plugin whose
attested behavior diverged from an unmodified one. Candidate and
unproven beyond that: whether behavioral attestation generalizes past
hooks (skills and agents are prose, with no test harness for their
behavior yet). Label any generalization claim open until one exists.
Inferred from the project's own research docs, and labeled as inference: the ambition is harness-level guardrails that keep autonomous loops honest and legible. The five problems above are one thread: gates the agent cannot fake (1), a skill library whose activation is measured rather than hoped (2), memory that survives resets and proves its retrieval (3), cross-plugin composition under hard budgets (4), and trust signals that cover behavior, not bytes (5). Advancing any one of them past its milestone is a contribution the wider agent-tooling field does not yet have.
docs/research/ and this file's entry was updated or removed.Compiled 2026-07-02 against repo v1.9.15 (branch discussions-fix-1.9.14). Volatile facts and how to re-verify them:
find plugins -name SKILL.md | wc -lrg -n "completion_integrity" plugins/egregore/scripts/config.pyrg -n "LLM_TIMEOUT_SECONDS" plugins/herald/hooks/double_shot_latte.pyhead -5 docs/specification.mdrg -n "_BUDGET_SECONDS" plugins/abstract/hooks/post_learnings_stop.pygh issue view 574 --json state -q .state (same for 553)rg -l "Approve with actions" plugins/pensive/skills/*/SKILL.md | wc -lrg -n "16K characters" docs/quality-gates.mdgit log --oneline -1 <hash>.Unverified in this compilation: the exact 16K-character discovery budget figure is the repo's own estimate ("about 16K characters" in docs/quality-gates.md), not an upstream-documented limit. The claim that no published activation-quality metric exists is a literature-absence claim as of 2026-07-02. Re-check before publishing externally.