abk-feedback

name

abk-feedback

description

Report an abkit bug, request a feature, or send feedback to the maintainers as a GitHub issue on the upstream repo (alexeiveselov92/ab-analysis-kit). Use when an `abk` command errors or behaves unexpectedly and it looks like an abkit defect (not the user's config), when the user wishes abkit did something it doesn't, or when they ask to report/file an issue, send feedback, or contact the maintainers. Auto-collects diagnostic context, redacts all secrets, and never submits without explicit confirmation.

Send feedback to the abkit maintainers

Turn a problem, a wish, or a comment into a clear GitHub issue on the upstream repo alexeiveselov92/ab-analysis-kit — carrying the diagnostic context maintainers need and leaking no secrets. You are filing a public issue on the user's behalf: never submit anything without showing the exact text and getting an explicit "yes".

This skill is the procedure; the reference for ruling out config problems lives under .claude/rules/ab-analysis-kit/.

Step 0 — Decide whether this belongs upstream

• Bug — an abk command errors or produces clearly wrong output, and you've ruled out a local config/usage mistake (Step 1).
• Feature request — the user needs abkit to do something it doesn't.
• Feedback / question — a comment, confusion, or rough edge worth surfacing.

Do not open an issue for the user's own SQL, data, database, or configuration mistakes — those are fixed locally with the rules files, not reported as abkit bugs. Only genuine abkit behavior goes upstream.

Step 1 — Rule out config first (for "it doesn't work")

Before calling something a bug, confirm it isn't a configuration or usage problem. Run the config lint — abk run --steps validate (this is the L1+L2 config validator; it is not abk validate, which is the A/A matrix). Then read the matching .claude/rules/ab-analysis-kit/ file and check the usual gotchas:

• every metric SQL is one row per unit, imports the cohort macro ({% import 'abkit_assignment.jinja' as ab %}) and joins ab.exposed_units(...) — don't hand-roll the assignment join;
• the metric type matches its columns (sample = value + optional CUPED covariate; fraction = count + nobs; ratio = numerator + denominator) — and the quantity is additive (sum/count), not a median/quantile;
• experiment and metric names are unique across the whole project (one namespace — the name is the DB key);
• profiles.yml has the right connection fields for the backend and a profile that exists, with secrets via ${ENV};
• SRM first — a blocked verdict with srm_flag set is a broken assignment / cohort query, not an abkit bug; fix the split before suspecting the effect;
• a withheld WIN/LOSE before the horizon is expected unless sequential: {enabled: true} is on (fixed-horizon CIs are not peeking-valid) — not a bug;
• retuned method_params orphan the prior series (new method_config_id); recompute + abk clean --select <exp> if stale rows are the confusion.

If a local fix exists, do that and stop — don't file an issue. Escalate only what is genuinely an abkit defect.

Step 2 — Gather context (do this for the user, don't make them dig)

Collect automatically:

• abk --version — the abkit version (always include it).
• Python version (python3 --version) and OS.
• The database backend type only (clickhouse / postgres / mysql) from profiles.yml's type: — never the host, port, or credentials.

For a bug, also gather:

• the exact command that was run (with its flags: --select / --metric / --method / --from / --to / --full-refresh etc.);
• the full error message / traceback;
• what the user expected vs. what happened;
• a minimal reproduction — the failing experiment and/or metric YAML and its SQL, with every secret stripped (Step 3).

Step 3 — Redact secrets (mandatory — before anything is shown or sent)

This is a public issue. Strip from every snippet you include:

• passwords, tokens, API keys, connection strings;
• hostnames, IPs, ports, and real database / schema / table names;
• any real user, unit, or experiment identifiers in sample data;
• the resolved value of anything inside ${…} — keep the ${ENV} shape, drop the value.

Replace with placeholders like <redacted>, <your-host>, <your_table> while keeping the structure intact so the bug is still reproducible. If you're not sure whether something is sensitive, redact it.

Step 4 — Search for duplicates

gh issue list --repo alexeiveselov92/ab-analysis-kit --search "<keywords>" --state all

If a matching issue already exists, offer to add a comment or a 👍 reaction instead of opening a new one:

gh issue comment <number> --repo alexeiveselov92/ab-analysis-kit --body "<note>"

This gives maintainers signal on frequency instead of a pile of duplicates.

Step 5 — Draft the issue

Write a specific title (the actual symptom in a few words, not "bug"). Pick the body template by type:

• Bug — Summary / Environment (abkit version, Python, OS, backend) / Steps to reproduce / Expected / Actual (with the traceback in a fenced code block) / Minimal config (redacted experiment + metric YAML/SQL).
• Feature request — Problem (the underlying need, not just the proposed fix) / Proposed behavior / Alternatives considered.
• Feedback — free-form, concrete and respectful.

End the body with an attribution marker so maintainers can see it came through the assistant funnel and on which version:

_Filed via the abk init-claude assistant (abkit <version>)._

Step 6 — Preview and confirm (no silent submits)

Show the user the full title and body exactly as they will be posted, and the target repo. Ask them to confirm: secrets redacted? right repo? Proceed to Step 7 only on an explicit "yes". If they want edits, revise and re-preview.

Step 7 — Submit

Preferred — gh CLI (check it's installed and authenticated first with gh auth status). Write the body to a temp file to avoid shell-escaping the markdown/traceback:

gh issue create --repo alexeiveselov92/ab-analysis-kit \
  --title "<title>" --body-file <tmpfile>

Tag the issue so maintainers can triage assistant-filed reports: add the type label (--label bug / --label enhancement) and --label "via:assistant". Labels must already exist on the repo, so if gh reports an unknown label, retry without the failing label(s) — never fail the report over a missing label. (Attribution also lives in the Step 5 body marker, so maintainers can filter with in:body "Filed via the abk init-claude assistant" even before the via:assistant label exists.) Return the issue URL to the user.

Fallback — no gh, or not authenticated. Build a prefilled "new issue" URL and hand it to the user to open and submit in their browser:

https://github.com/alexeiveselov92/ab-analysis-kit/issues/new?title=<url-encoded>&body=<url-encoded>

URL-encode both fields. If the body is too long for a URL, give the user the title plus the body text to paste, and the plain link https://github.com/alexeiveselov92/ab-analysis-kit/issues/new.

Step 8 — Close the loop

Give the user the issue URL (or the prefilled link), mention they can subscribe for updates, and — if you also worked around the bug locally — restate that local fix so they're unblocked right now, not just waiting on the issue.