next-best-questions

name	next-best-questions
description	Use when a problem or request is underspecified and you need to decide WHAT to clarify before doing the work — the NEXT-BEST QUESTIONS to answer, ranked. Interrogates the prompt into candidate questions, projects plausible answers with probabilities, estimates each question's value of information (how much the answer would change the recommended plan, weighted by likelihood and stakes), discards low-value/redundant ones, and keeps generating until a diverse bucket of high-value questions is filled. Reports a ranked list with recommendations (pre-answer / assume-default) using role-specialized local Ollama models. Reports only — it does not ask the user or answer the questions itself. Triggers: "what should I clarify", "what questions matter here", "is this spec complete", "what am I missing before I start".
version	1.3.8
author	agent
license	MIT
platforms	["linux","macos","windows"]
metadata	{"hermes":{"category":"autonomous-ai-agents","tags":["information-gain","value-of-information","evsi","clarifying-questions","planning","decision-support","ollama"],"related_skills":["ask","advisors"],"config":[{"key":"next-best-questions.question_gen_model","description":"Model alias that generates and frames candidate questions (strong)","default":"glm","prompt":"Which model should generate questions for next-best-questions?"},{"key":"next-best-questions.answer_model","description":"Fast model alias that projects plausible answers (runs in parallel)","default":"fast","prompt":"Which fast model should project answers?"},{"key":"next-best-questions.value_judge_model","description":"Model alias that judges per-answer plan-change and stakes (strong)","default":"deepseek","prompt":"Which model should judge question value?"},{"key":"next-best-questions.min_bucket_size","description":"Minimum number of high-value questions the report aims to contain","default":3,"prompt":"Minimum bucket size for ranked questions?"},{"key":"next-best-questions.discard_threshold","description":"Value (0-1) below which a question is dropped as not valuable","default":0.3,"prompt":"Discard threshold for low-value questions?"},{"key":"next-best-questions.max_rounds","description":"Max generation rounds while trying to fill the bucket","default":3,"prompt":"Max generation rounds?"}]}}

Next-Best-Questions — what to clarify before you start

See ../nbq-improve/SKILL.md for the improvement protocol for this skill.

Overview

Given an underspecified problem, this skill estimates the value of information of clarifying questions and reports the ones worth resolving first. It approximates the decision-theoretic Expected Value of Sample Information (EVSI): a question is valuable only if its answer is genuinely uncertain and would change the recommended plan and the stakes of guessing wrong are real. It simulates plausible answers with local Ollama models, scores each question, discards the low-value and redundant ones, and keeps generating until a diverse bucket is filled.

It is a reporter / analysis primitive — it ranks and recommends. Deciding to ask the user, or researching an answer, is the caller's job (same primitive-vs-orchestrator discipline as ask).

When to Use

Use it before committing to an approach on a vague brief, ambiguous ticket, or open-ended ask — to surface the high-leverage unknowns and a safe default for each.

Don't use it for well-specified tasks (it will correctly report "nothing high-value to clarify" and you've spent model calls to learn that), or when you just want a single model's opinion (use ask), or for a multi-model debate (use advisors).

How to run

The logic is in scripts/; run it with the terminal tool. Skill dir: ${HERMES_SKILL_DIR}.

# Markdown report (default = focus mode: the prioritized top few)
python3 ${HERMES_SKILL_DIR}/scripts/infogain.py "Build a service to sync USAW events into our calendar"

# Breadth mode: wider coverage by SAMPLING the model's own question distribution
# (several high-temperature draws, unioned + deduped) — no seeded topic list.
# Heavier/slower — pair with a fast judge if needed.
python3 ${HERMES_SKILL_DIR}/scripts/infogain.py "<problem>" --mode breadth

# Structured JSON for programmatic use (read the bucket back into your reasoning)
python3 ${HERMES_SKILL_DIR}/scripts/infogain.py -p "Add SSO to the admin portal" --json

# See the exact stage prompts without any model calls
python3 ${HERMES_SKILL_DIR}/scripts/infogain.py "<problem>" --dry-run

# Show your work: every stage's prompt + raw model output + the per-question
# scoring arithmetic (U, the P·Δplan·stakes terms, √(U·EVSI)) + the loop decisions.
# Invaluable for comparing models or scaffolding a weaker one. Add --json for structured.
python3 ${HERMES_SKILL_DIR}/scripts/infogain.py "<problem>" --trace

# Iterative loop: fold answered evidence back into the same context; resolved questions drop
# out (they become derivable) and the next-best questions surface. You/the agent drive the loop.
python3 ${HERMES_SKILL_DIR}/scripts/infogain.py "<problem>" \
    --evidence "data residency: US-only, no third-party cloud" "budget: free tier only"
# ...or --evidence-file facts.txt (one established fact per line).

# Write to a file; quiet stderr
python3 ${HERMES_SKILL_DIR}/scripts/infogain.py "<problem>" -o /tmp/infogain.md --quiet

Use --problem/-p (not positional) when the text contains -- or shell-special characters.

What it returns

Given a prompt, a list of the key questions ranked by weight = exploration value = √(uncertainty × value-of-answering) — how much answering each would improve your response to the prompt. Each ranked question carries a plain-language clarification of what its weight means (how much it would improve the response, and the assumption you'd otherwise make and its chance of being wrong), plus the recommendation (PRE_ANSWER ≥ 0.60 / ASSUME_DEFAULT ≥ 0.30). A detailed numeric table is included below the list. If nothing clears the bar, it says so — the prompt is already specified well enough for a good response. See references/design-decisions.md for the model.

Example (abridged)

$ python3 scripts/infogain.py "Add authentication to my web app."

Key questions, ranked by how much answering them improves the response:

1. (0.74, PRE_ANSWER) What kind of users authenticate — your own customers, or do you
   need social/enterprise SSO (Google, SAML)?    [family: Auth strategy]
   Without it I'd assume email+password sessions; ~60% chance that's not what you need,
   and the choice reshapes the whole design.
2. (0.66, PRE_ANSWER) What is the app's stack (framework, existing user table/DB)?
   [family: Stack & integration points]
   I'd otherwise pick a generic middleware answer that may not drop into your codebase.
3. (0.48, ASSUME_DEFAULT) Buy vs build — is a hosted provider (Auth0/Clerk/Cognito)
   acceptable, or must this be self-hosted?    [family: Should we build this? — contrarian]
   Default: self-hosted library; flag if compliance or pricing forces the other path.
...
(numeric U / EVSI / value table follows)

Answer one of these (yourself or via research), re-run with --evidence "stack -> Django, existing users table", and the resolved question drops out while the next-best surfaces.

How it works (4 stages, looped)

1. Frame + baseline plan (plan_model) — restate goal/decision and the plan you'd give right now (folding in any --evidence); everything is scored as change from this baseline.
2. Project answers (answer_model, parallel) — plausible answers + probabilities, plus how derivable each question is (already inferable from the prompt?).
3. Judge (value_judge_model, parallel) — per-answer plan-change × stakes vs the baseline.
4. Score / gate / diversify (pure Python) — value-of-answering (EVSI) = Σ P·Δplan·stakes; uncertainty U = entropy(answers)·(1−derivable); gate out the no-uncertainty/no-change cases; exploration value = √(U · EVSI); collapse same-target duplicates; MMR-rank. If the bucket is under min_bucket_size, generate another round (deduped) up to max_rounds.

Full rationale + citations: references/methodology.md. Prompt contracts: references/prompts.md.

Families (default ON — --families/--no-families)

Before generating individual questions, it first generates families of questions for coverage: several scoped families (distinct regions of the unknowns) plus three lenses you'd otherwise miss — a contrarian family (challenges the baseline approach itself: "should we even build this?"), a vantage family (questions whose answer is access-relative — differs by environment / server / identity / credential / token; auto-enabled only for systems/access tasks), and a pre-mortem family (assume the plan shipped and FAILED — what unknown would have prevented it?: data loss, security compromise, irreversible/destructive actions, silent wrong output, runaway cost; auto-enabled only for failure-surface tasks — writes/deploys/payments/migrations), and a reach family (#29: does a DIFFERENT, REACHABLE point of view exist — a container you can enter, a host you can SSH to, a service you can execute inside, possibly via CHAINED hops — that would turn an unknown into an observable?; shares the vantage auto-gate; --reach on|off|auto / INFOGAIN_REACH; the ranker surfaces the hop, the investigator executes it). It then generates questions within each family (tagged with family/lens).

The non-scoped lenses map onto the value formula √(U·EVSI), EVSI = Σ P·Δplan·stakes: scoped maximizes Δplan-coverage, contrarian targets the premise (highest Δplan), vantage targets a source of hidden Δplan, reach hunts answerability (what could become derivable from another vantage), and pre-mortem hunts the stakes tail — the catastrophic/irreversible branch no other lens systematically surfaces. (Pre-mortem is the generation-side, formula-frozen half of the deferred "risk-averse tilt": it only ensures the catastrophic-tail question enters the candidate set; scoring stays risk-neutral, so a lurid-but-improbable question still self-prunes on low P.)

Families are domain exposure only — there is no family-level negation. Every question is still scored on its own merit (the same EVSI pipeline), so a low-average family can still surface the single highest-value question (this is common for the contrarian/vantage/pre-mortem lenses); irrelevant families self-prune because their questions score low individually. The family tag adds a tier to the MMR diversity kernel (family > target > question) so selection spreads across families, and the report is grouped by family (non-scoped lenses labelled). Turn off with --no-families (or INFOGAIN_FAMILIES=off) for the flat generator; force the pre-mortem lens with --premortem on|off|auto (or INFOGAIN_PREMORTEM). The families layer runs on its own model (families_model, default glm — not covered by --question-gen-model); pin it with --families-model <alias> (or INFOGAIN_FAMILIES_MODEL) for offline/reproducible runs. Cost ≈ questions_per_family × (n_scoped + contrarian + vantage + premortem + reach) candidates (~12–18 at defaults), with a one-call fallback to the flat generator if family generation yields nothing.

Automating the loop — the investigator skill

This skill is report-only — it ranks what to clarify and stops. To automate the evidence loop (research the top questions, fold facts back in, re-rank, and produce a final response), use the sibling investigator skill (autonomous-ai-agents/investigator/), which calls this ranker and owns the answering loop + a capability ladder. Keeping the answering/looping out of here preserves the primitive-vs-orchestrator boundary. See that skill's SKILL.md.

Tuning

Mode presets: --mode focus (default — prioritized top few) or --mode breadth (wider coverage). Breadth works by sampling the model's own distribution over questions — gen_samples independent draws at gen_temperature, unioned and deduped — so the breadth comes from the model's uncertainty (the tail of its distribution), not a seeded topic list. Scoring stages stay deterministic (temperature 0): explore stochastically, evaluate stably. When gen_samples > 1 the sampled candidates are then semantically consolidated — a consolidate_model (default fast) clusters questions that resolve the same underlying unknown and keeps one canonical per cluster, so high-temperature sampling doesn't pad the bucket with reworded duplicates (it never drops a distinct unknown; on failure it falls back to the lexically-deduped set). Resolution order is DEFAULTS ← mode preset ← INFOGAIN_* env ← CLI flag, so a preset sets the baseline and any explicit flag/env still wins (e.g. --gen-samples 5 --gen-temperature 1.0).

Defaults live as module constants in scripts/infogain.py, overridable by INFOGAIN_* env vars or CLI flags (e.g. --min-bucket-size 5, --discard-threshold 0.5, --answer-model qwen). Model names are ask-style aliases (glm, deepseek, fast, qwen, …) resolved via model_utils. --value-judge-mode absolute|pairwise|solution|behavior selects the Δplan/stakes elicitation (pairwise = forced-choice → Bradley-Terry, the #24 experiment — its powered A/B closed keep absolute, so the flag exists for re-testing, not for live use; solution = Δplan grounded as the fraction of --solution-samples sampled candidate solutions an answer invalidates — the #27 experiment; its powered A/B closed keep absolute, decisively: solution deltas collapse to near-binary and within-task ρ goes negative; behavior = Δplan as BEHAVIOR/outcome change of the delivered result, consequence not code size — the #28 experiment; its OBJECTIVE gate closed keep absolute: directionally better but 6W/5L vs the required broad win, and the naive baseline still out-asks it — the residual gap is generation altitude, not judging). --answer-prob-mode stated|sampled selects the P(a) estimate (sampled = Laplace-smoothed frequencies over --answer-samples forced-choice draws instead of the model's self-stated probabilities — the #26 experiment; its powered A/B closed keep stated (a real-contrast null: P moved on 79% of pairs, ranking didn't improve), so the flag exists for re-testing, not for live use; stated probs survive as stated_prob).

Derive-or-ask (default ON since 1.3.0): a question whose answer the model claims it basically knows (derivable_prob ≥ --derive-threshold, default 0.6) is tested, not trusted — a derivation attempt (--derive-model, default = the value-judge model) either states the answer, which is tombstoned into the working context and reported under "Resolved during analysis — treated as evidence" (it was never really a question), or replies CANNOT_DERIVE, in which case the inflated claim is corrected (--cannot-derive-cap, default 0.2) and the question re-enters ranking with its uncertainty honestly restored. Refill rounds re-plan against the tombstones; a derivation landing in the final round grants one bounded extra round. Capped at --derive-max-per-round (default 4) attempts. Rollback: --auto-derive off / INFOGAIN_AUTO_DERIVE=off (pre-1.3.0 behavior: claims silently trusted, derivable questions suppressed without being answered). Eval-harness cfgs built from DEFAULTS (no key) stay OFF for comparability with the powered datasets.

Dependency

Reuses the ask skill's dispatch helpers (model_utils.py). The scripts resolve it at runtime via HERMES_HOME (default ~/.hermes → /opt/data in-container) or an explicit ASK_SCRIPTS_DIR. If the ask skill isn't installed, the scripts exit with a clear message.

Common Pitfalls

• Ollama must be reachable at host.docker.internal:11434 (override OLLAMA_URL). The script preflights /api/tags and exits 2 with a clear message if it's down.
• glm defaults to Chinese — handled automatically (build_prompt appends an English directive for NON_ENGLISH_MODELS); don't hand-roll one.
• It reports, it doesn't act. Don't expect it to ask the user or fill in answers — feed its shortlist into your own clarify/research step.
• A short or empty bucket is a valid result, not a failure — the problem is well-specified.
• Cost scales with rounds × questions × 2 calls. Cloud judge/gen models add latency; switch to local aliases (--question-gen-model qwen --value-judge-model qwen) for fast/cheap runs.

Evaluating output quality (adjudicated evals)

evals/ holds an adjudicator harness that decides whether the skill's output is acceptable, not just well-formed. evals/run_evals.py runs the skill on evals/cases.json (a mix of underspecified and well-specified problems) and applies two layers:

• structural checks (deterministic) — values in [0,1], ranked order, valid recommendations, no surviving duplicate targets, and per-case calibration (an underspecified problem must yield several questions; a well-specified one must yield few/none).
• an LLM adjudicator (evals/adjudicator.py) — a different, stronger model than the generation stages (default deepseek judging fast-generated output, to avoid self-judging bias) scores framing accuracy, question relevance, value justification, diversity, and calibration.

# all cases (exit non-zero if any unacceptable — usable as a CI gate)
python3 ${HERMES_SKILL_DIR}/evals/run_evals.py --gen-model fast --judge-model deepseek
python3 ${HERMES_SKILL_DIR}/evals/run_evals.py --case reverse-string --json

Weak runs are expected to fail (e.g. fast + 1 round on a hard problem); stronger generation (--gen-model glm --max-rounds 2) should clear the bar. That two-sided behavior is the point.

Verification Checklist

• python3 -m py_compile scripts/*.py evals/*.py passes.
• uv run --with pytest python3 -m pytest tests/ -v -k "not live" is green (pure logic + adjudicator logic).
• --dry-run prints all four stage prompts (confirms model_utils import + builders).
• With Ollama up, a live run on a vague problem returns a ranked bucket and a "pre-answer" list.
• A deliberately well-specified problem yields a small/empty bucket with the "well-specified" note.
• evals/run_evals.py rates the well-specified case acceptable (bucket≈0) and flags shallow runs.
• Frontmatter validates (starts at byte 0 with ---, has name + description).