authoring-gzcli-challenge-quality

name	authoring-gzcli-challenge-quality
description	Use this skill to audit the quality of a finished gzcli challenge before sync / upload. Covers the four pillars — no-guessing / under-specified design, no unintended solutions that bypass the intended chain, no hints leaked in source or shipped artifacts, and the wider quality checklist (reproducibility, leak-free shipping, difficulty calibration, container hygiene).

Challenge Quality Audit

A pre-ship checklist for gzcli challenges. Run through this before treating a challenge as ready. Each section names a failure category to actively probe for. For per-folder authoring rules, defer to the matching sub-skill (-yml, -src, -dist, -solver).

The four pillars

1. No guessing

A quality challenge has one obvious next step at every stage. Players reason forward; they do not brute-force or guess.

Symptoms of a guessing challenge:

• The description gives no concrete hook ("find the flag in this binary" with nothing else). Players try every tool randomly.
• The solve requires a specific constant, key, or transformation that isn't derivable from the artifact (e.g. "XOR with 0x42" without any signal that 0x42 is the key).
• Multiple plausible techniques work to varying degrees, and which one is correct isn't observable until you try them all.
• The challenge depends on a private wordlist, dictionary, or trivia the player can't reasonably possess.
• The flag itself is something the player has to guess (specific quote, song lyric, joke).

How to test: read your own challenge as a stranger. From the description + artifacts alone, can you derive the next step? If you have to "remember" something from authoring, it's a guess. Fix by surfacing the necessary signal in the artifact (leave the constant discoverable in the binary; structure the file format so the technique is implied).

2. No unintended solutions

The flag must be reachable only via the intended chain. Probe for shortcuts in every category:

• Source / attachment greps. strings dist/* and grep -rE '<PREFIX>\{[^}]+\}' dist/. The real flag must not appear. Decoys per the dist sub-skill's decoy-resistance rule are OK (and required for forensics).
• Web sub-resource shortcuts. Probe /admin, /debug, /console, /.env, /.git/HEAD, /robots.txt, /sitemap.xml, /server-status, /phpinfo.php. Check for directory listing on /static/, /assets/. Check for source maps (*.map), backup files (*.bak, *~).
• Auth / session shortcuts. For admin-bot challenges: can the player exfiltrate without traversing the intended chain (e.g. inline <script> if CSP isn't set, open redirect that obviates CSRF, missing Sec-Fetch-Site check)?
• Container surface. docker exec <container> ls -la /flag.txt — world-readable when the intended chain is "exploit gives RCE → reads flag"? docker history <image> — any ARG/COPY layer leaking secrets? docker inspect — stray volume mounts? Unintended ports?
• Reverse-engineering shortcuts. file binary — stripped? strings -n 4 binary | grep -iE 'flag|<PREFIX>' — too revealing? ltrace/strace reveals the comparison in cleartext? Debug symbols (-g) left in?
• Pwn shortcuts. checksec — does protection level match stated difficulty? Hidden system() callable, stray gets(), world-writable /proc/self/environ?
• Crypto shortcuts. Is n factor-able by FactorDB? IV reused across messages? Key shipped alongside ciphertext? Padding-oracle when the intended chain is something else?
• Forensics shortcuts. strings disk.img | grep '<PREFIX>{', binwalk -e, photorec, read-only mount — none must yield the real flag.
• Solver / writeup divergence. If solver/solve.py takes a different path from solver/README.md, the author probably found a shortcut mid-development. Audit it.

The intended chain must be the only path under reasonable effort.

3. No hints leaked in source or attachments

Source under src/ (rebuilt into the running container) and source shipped via dist/ must read like normal application code. Players learn the technique by exploring, not by reading authorial tour guides.

Audit checklist:

• Comments that name the vulnerability: # format string bug here, // XOR with 0x37, // TODO: fix SQLi in /login, # debug backdoor.
• Identifiers that name the answer: char flag[], void win(), int secret_xor_key, bool is_admin_bypass, fn unsafe_eval(...). Rename to neutral: session_token, admin_dump_motd, _0xK.
• Debug modes leaking state: FLASK_DEBUG=1, DJANGO_DEBUG=True, NODE_ENV=development, APP_DEBUG=true, verbose error pages.
• Local-test flag literals in dist/ that use the event prefix — must be fake{flag} (see authoring-gzcli-challenge-dist).
• Authorial commentary that wouldn't appear in production code ("this is the vulnerable function", "intended path: XSS via /preview").

One-shot grep:

grep -rEni 'TODO|FIXME|XXX|HACK|vuln|exploit|backdoor|bypass|intended|spoiler' src/ dist/

Every hit must be defensible as "normal application code".

Description-side hint leaks are handled by authoring-gzcli-challenge-yml's "Description style" rule. Cross-reference that for any **Hint:** blocks, leading questions, or "the flag lives in X" reveals.

4. Wider quality

Beyond the three pillars:

• Reproducibility. Build twice from a clean checkout; resulting artifacts match (modulo intentional random padding). Solver succeeds against the just-built challenge, not a cached image. Add docker rmi <slug>:latest before re-running the validation script.
• Solver runs end-to-end. python3 solver/solve.py exits 0 and prints a regex-matchable flag against a freshly-built container.
• Difficulty calibration matches the description. "Easy" challenges shouldn't need niche tooling; "medium" shouldn't be strings-solvable.
• Time budget. Solver succeeds in seconds-to-a-few-minutes once the chain is understood. Multi-hour brute-force isn't difficulty, it's tedium.
• Container hygiene. Only exposePort: is reachable. No SSH backdoor, no debug interpreter, no --privileged. Unprivileged user where the listener doesn't need root.
• Resource limits sane. memoryLimit / cpuCount / storageLimit in challenge.yml cover the actual workload — container doesn't OOM during normal use.
• Per-team isolation. DynamicContainer challenges don't share state across teams via persistent files or external services. Per-team flag comes from GZCTF_FLAG.
• Failure-mode debuggability. Failed solver attempts yield observable signals (HTTP status, response body, container log) — not silent timeouts or generic 500s.
• Network determinism. No external DNS / third-party APIs in the intended chain — those flake during real events. Self-contained.
• Race-condition-free. The intended chain succeeds deterministically. If timing matters, the solver has retry / wait logic.
• No 0-day or unpublished techniques. For educational events, the technique should be publicly documented (writeups, papers, MDN, OWASP). Players shouldn't need insider knowledge.
• Solver / writeup parity. solver/solve.py and solver/README.md describe the same chain.
• Per-team flag flavor doesn't telegraph. A flag like LKS{csrf_pwns_the_post_xss} is fine after a solve (helps recap), but doesn't appear anywhere the player can read it before solving.

Pre-ship checklist (one-pass)

Walk through this before merging:

1. Solver runs clean. docker rmi <slug>:latest 2>/dev/null; cd src && docker build -t <slug>:latest .; docker run …; python3 ../solver/solve.py … — exits 0, prints the flag.
2. No real-flag leak in dist/. grep -rE '<PREFIX>\{[^}]+\}' dist/ returns only decoys (or nothing).
3. No spoiler text in source. grep -rEni 'TODO|FIXME|vuln|exploit|backdoor|intended|spoiler' src/ dist/ — every hit defensible.
4. Web probes. curl -sI http://target/, curl http://target/{flag,/flag.txt,/.env,/.git/HEAD,/admin,/debug,/console}. Nothing reveals the flag or skips steps.
5. Pwn checksec. Matches stated difficulty. strings binary | grep -iE 'flag|<PREFIX>' returns nothing useful.
6. Forensics strings. strings dist/* | grep '<PREFIX>{' returns only decoys (or nothing).
7. Crypto algebraic shortcut audit. The "obvious" alternative attack isn't easier than the intended one.
8. Container clean. Starts within a few seconds, exposes only exposePort:, doesn't OOM under normal use, docker history clean.
9. Description. No **Hint:** lines, English-only, neutral CTA, no telegraphing questions.
10. challenge.yml schema. value: 1000. containerImage: "{{.slug}}:latest". scripts.start: cd src && docker build -t {{.slug}} .. flagTemplate: has exactly one trailing }.
11. dist/ flag literals. If source is shipped, fallback literals are fake{flag}, not event-prefixed.
12. Solver / writeup parity. Same chain in both solver/solve.py and solver/README.md.

Any failed item is a blocker — fix before shipping.

Brainstorm prompts when designing a new challenge

Run these mental checks during design, not just before shipping:

• "What's the worst (laziest) tool a player could try first?" If that tool yields the flag, the challenge is broken.
• "If I gave a fresh player only the description and the artifacts, would they know what category of technique to try?" If no, you're under-specifying.
• "If a player solves the challenge in 30 seconds, is that because they're skilled or because there's an unintended path?" Test by handing it to a teammate cold.
• "What does the solver look like in 50 lines or fewer?" If you can't sketch it that compact, the chain is too complicated.
• "What does the writeup look like in 200 words?" If you can't summarize the technique briefly, you may be combining two challenges.

Related sub-skills

• authoring-gzcli-challenge-yml — description-style and value rules cited above.
• authoring-gzcli-challenge-src — source comment / identifier rules.
• authoring-gzcli-challenge-dist — decoy-resistance and flag-placeholder rules.
• authoring-gzcli-challenge-solver — solver clean-start validation.
• authoring-gzcli-ctf-events — event-level orchestrator.