| name | authoring-gzcli-challenge-quality |
| description | Use this skill to audit the quality of a finished gzcli challenge before sync / upload. Covers the four pillars — no-guessing / under-specified design, no unintended solutions that bypass the intended chain, no hints leaked in source or shipped artifacts, and the wider quality checklist (reproducibility, leak-free shipping, difficulty calibration, container hygiene). |
Challenge Quality Audit
A pre-ship checklist for gzcli challenges. Run through this before treating a challenge as ready. Each section names a failure category to actively probe for. For per-folder authoring rules, defer to the matching sub-skill (-yml, -src, -dist, -solver).
The four pillars
1. No guessing
A quality challenge has one obvious next step at every stage. Players reason forward; they do not brute-force or guess.
Symptoms of a guessing challenge:
- The description gives no concrete hook ("find the flag in this binary" with nothing else). Players try every tool randomly.
- The solve requires a specific constant, key, or transformation that isn't derivable from the artifact (e.g. "XOR with 0x42" without any signal that 0x42 is the key).
- Multiple plausible techniques work to varying degrees, and which one is correct isn't observable until you try them all.
- The challenge depends on a private wordlist, dictionary, or trivia the player can't reasonably possess.
- The flag itself is something the player has to guess (specific quote, song lyric, joke).
How to test: read your own challenge as a stranger. From the description + artifacts alone, can you derive the next step? If you have to "remember" something from authoring, it's a guess. Fix by surfacing the necessary signal in the artifact (leave the constant discoverable in the binary; structure the file format so the technique is implied).
2. No unintended solutions
The flag must be reachable only via the intended chain. Probe for shortcuts in every category:
- Source / attachment greps.
strings dist/* and grep -rE '<PREFIX>\{[^}]+\}' dist/. The real flag must not appear. Decoys per the dist sub-skill's decoy-resistance rule are OK (and required for forensics).
- Web sub-resource shortcuts. Probe
/admin, /debug, /console, /.env, /.git/HEAD, /robots.txt, /sitemap.xml, /server-status, /phpinfo.php. Check for directory listing on /static/, /assets/. Check for source maps (*.map), backup files (*.bak, *~).
- Auth / session shortcuts. For admin-bot challenges: can the player exfiltrate without traversing the intended chain (e.g. inline
<script> if CSP isn't set, open redirect that obviates CSRF, missing Sec-Fetch-Site check)?
- Container surface.
docker exec <container> ls -la /flag.txt — world-readable when the intended chain is "exploit gives RCE → reads flag"? docker history <image> — any ARG/COPY layer leaking secrets? docker inspect — stray volume mounts? Unintended ports?
- Reverse-engineering shortcuts.
file binary — stripped? strings -n 4 binary | grep -iE 'flag|<PREFIX>' — too revealing? ltrace/strace reveals the comparison in cleartext? Debug symbols (-g) left in?
- Pwn shortcuts.
checksec — does protection level match stated difficulty? Hidden system() callable, stray gets(), world-writable /proc/self/environ?
- Crypto shortcuts. Is
n factor-able by FactorDB? IV reused across messages? Key shipped alongside ciphertext? Padding-oracle when the intended chain is something else?
- Forensics shortcuts.
strings disk.img | grep '<PREFIX>{', binwalk -e, photorec, read-only mount — none must yield the real flag.
- Solver / writeup divergence. If
solver/solve.py takes a different path from solver/README.md, the author probably found a shortcut mid-development. Audit it.
The intended chain must be the only path under reasonable effort.
3. No hints leaked in source or attachments
Source under src/ (rebuilt into the running container) and source shipped via dist/ must read like normal application code. Players learn the technique by exploring, not by reading authorial tour guides.
Audit checklist:
- Comments that name the vulnerability:
# format string bug here, // XOR with 0x37, // TODO: fix SQLi in /login, # debug backdoor.
- Identifiers that name the answer:
char flag[], void win(), int secret_xor_key, bool is_admin_bypass, fn unsafe_eval(...). Rename to neutral: session_token, admin_dump_motd, _0xK.
- Debug modes leaking state:
FLASK_DEBUG=1, DJANGO_DEBUG=True, NODE_ENV=development, APP_DEBUG=true, verbose error pages.
- Local-test flag literals in
dist/ that use the event prefix — must be fake{flag} (see authoring-gzcli-challenge-dist).
- Authorial commentary that wouldn't appear in production code ("this is the vulnerable function", "intended path: XSS via /preview").
One-shot grep:
grep -rEni 'TODO|FIXME|XXX|HACK|vuln|exploit|backdoor|bypass|intended|spoiler' src/ dist/
Every hit must be defensible as "normal application code".
Description-side hint leaks are handled by authoring-gzcli-challenge-yml's "Description style" rule. Cross-reference that for any **Hint:** blocks, leading questions, or "the flag lives in X" reveals.
4. Wider quality
Beyond the three pillars:
- Reproducibility. Build twice from a clean checkout; resulting artifacts match (modulo intentional random padding). Solver succeeds against the just-built challenge, not a cached image. Add
docker rmi <slug>:latest before re-running the validation script.
- Solver runs end-to-end.
python3 solver/solve.py exits 0 and prints a regex-matchable flag against a freshly-built container.
- Difficulty calibration matches the description. "Easy" challenges shouldn't need niche tooling; "medium" shouldn't be
strings-solvable.
- Time budget. Solver succeeds in seconds-to-a-few-minutes once the chain is understood. Multi-hour brute-force isn't difficulty, it's tedium.
- Container hygiene. Only
exposePort: is reachable. No SSH backdoor, no debug interpreter, no --privileged. Unprivileged user where the listener doesn't need root.
- Resource limits sane.
memoryLimit / cpuCount / storageLimit in challenge.yml cover the actual workload — container doesn't OOM during normal use.
- Per-team isolation. DynamicContainer challenges don't share state across teams via persistent files or external services. Per-team flag comes from
GZCTF_FLAG.
- Failure-mode debuggability. Failed solver attempts yield observable signals (HTTP status, response body, container log) — not silent timeouts or generic 500s.
- Network determinism. No external DNS / third-party APIs in the intended chain — those flake during real events. Self-contained.
- Race-condition-free. The intended chain succeeds deterministically. If timing matters, the solver has retry / wait logic.
- No 0-day or unpublished techniques. For educational events, the technique should be publicly documented (writeups, papers, MDN, OWASP). Players shouldn't need insider knowledge.
- Solver / writeup parity.
solver/solve.py and solver/README.md describe the same chain.
- Per-team flag flavor doesn't telegraph. A flag like
LKS{csrf_pwns_the_post_xss} is fine after a solve (helps recap), but doesn't appear anywhere the player can read it before solving.
Pre-ship checklist (one-pass)
Walk through this before merging:
- Solver runs clean.
docker rmi <slug>:latest 2>/dev/null; cd src && docker build -t <slug>:latest .; docker run …; python3 ../solver/solve.py … — exits 0, prints the flag.
- No real-flag leak in
dist/. grep -rE '<PREFIX>\{[^}]+\}' dist/ returns only decoys (or nothing).
- No spoiler text in source.
grep -rEni 'TODO|FIXME|vuln|exploit|backdoor|intended|spoiler' src/ dist/ — every hit defensible.
- Web probes.
curl -sI http://target/, curl http://target/{flag,/flag.txt,/.env,/.git/HEAD,/admin,/debug,/console}. Nothing reveals the flag or skips steps.
- Pwn
checksec. Matches stated difficulty. strings binary | grep -iE 'flag|<PREFIX>' returns nothing useful.
- Forensics
strings. strings dist/* | grep '<PREFIX>{' returns only decoys (or nothing).
- Crypto algebraic shortcut audit. The "obvious" alternative attack isn't easier than the intended one.
- Container clean. Starts within a few seconds, exposes only
exposePort:, doesn't OOM under normal use, docker history clean.
- Description. No
**Hint:** lines, English-only, neutral CTA, no telegraphing questions.
challenge.yml schema. value: 1000. containerImage: "{{.slug}}:latest". scripts.start: cd src && docker build -t {{.slug}} .. flagTemplate: has exactly one trailing }.dist/ flag literals. If source is shipped, fallback literals are fake{flag}, not event-prefixed.- Solver / writeup parity. Same chain in both
solver/solve.py and solver/README.md.
Any failed item is a blocker — fix before shipping.
Brainstorm prompts when designing a new challenge
Run these mental checks during design, not just before shipping:
- "What's the worst (laziest) tool a player could try first?" If that tool yields the flag, the challenge is broken.
- "If I gave a fresh player only the description and the artifacts, would they know what category of technique to try?" If no, you're under-specifying.
- "If a player solves the challenge in 30 seconds, is that because they're skilled or because there's an unintended path?" Test by handing it to a teammate cold.
- "What does the solver look like in 50 lines or fewer?" If you can't sketch it that compact, the chain is too complicated.
- "What does the writeup look like in 200 words?" If you can't summarize the technique briefly, you may be combining two challenges.
Related sub-skills
authoring-gzcli-challenge-yml — description-style and value rules cited above.authoring-gzcli-challenge-src — source comment / identifier rules.authoring-gzcli-challenge-dist — decoy-resistance and flag-placeholder rules.authoring-gzcli-challenge-solver — solver clean-start validation.authoring-gzcli-ctf-events — event-level orchestrator.
