uft-protection-db

name

uft-protection-db

description

Use when adding, removing, or auditing entries in UFT's copy-protection title databases (`src/protection/c64/c64_protection_db.c`, `src/protection/uft_protection_classify.c`, `uft_protection_extended.c`). Trigger phrases: "protection db", "known title", "neuer kopierschutz eintrag", "c64 protection", "amiga protection", "duplicate title", "title lookup", "substring shadow bug". Captures the title-lookup contract (substring + bidirectional `strstr` → first-match-wins) and the de-dup invariant that was violated in v4.1.5 (9 stale duplicates shadowed canonical entries). DO NOT use for: protection-detection *algorithm* (→ `uft-protection-scheme`), generic database refactor (this skill is data-curation only), ML classifier training data (→ `uft-ml-training-data`).

UFT Protection Database

Use this skill when curating the static protection-title databases. These DBs map disk-name strings to known copy-protection schemes, version, year, and notes. A wrong entry produces a wrong forensic verdict on a real disk — and forensic verdicts are what the tool is for.

This skill exists because of a class of bugs that took 9 commits to fully unmask in v4.1.5 (see git log --grep="duplicate-title"): the lookup function is permissive, so a stale duplicate near the top of the table silently shadows the correct, canonical entry far below.

When to use this skill

• Adding a new entry to a protection DB
• Removing or merging duplicate entries (9 known cases as of v4.1.5, ~60 latent — see follow-up below)
• Auditing a DB for consistency after a large data ingestion
• Adding a new platform DB (e.g. spectrum, amstrad protections)
• Re-checking lookup behaviour after a refactor

Do NOT use this skill for:

• Implementing a protection-detection algorithm (signature matching, flux-pattern analysis) — use uft-protection-scheme
• Replacing the lookup function — that's a public-ABI change, requires abi-bomb-detector first
• ML-classifier training-data assembly — use uft-ml-training-data
• Filesystem-level "is this a copy-protected file?" — different layer

The lookup contract (read before editing)

/* src/protection/c64/c64_protection_db.c:1292 — canonical lookup */
bool c64_lookup_title(const char *title, c64_known_title_t *entry) {
    /* lowercase both sides */
    /* iterate g_known_titles in declaration order */
    if (strstr(lower_known, lower_title) ||      /* DB ⊇ query */
        strstr(lower_title,  lower_known)) {      /* query ⊇ DB */
        return *entry = g_known_titles[i], true;  /* FIRST match wins */
    }
}

Three properties matter — internalize them:

1. Bidirectional substring. "Bruce Lee" matches both "Bruce Lee II" (DB ⊇ query) and "Bruce" (query ⊇ DB).
2. First-match-wins. Iteration order = declaration order in g_known_titles[].
3. Lowercase, no Unicode normalization. Diacritics, full-width characters, non-ASCII punctuation aren't normalized.

Consequence: if entry "Bruce Lee" (line 38) and entry "Bruce Lee (Datasoft, US)" (line 700) both exist, the line-38 entry always wins for any query containing "bruce lee", even when the caller wanted the specific Datasoft variant. That is the shadow bug.

The 5 de-dup invariants (hard)

#	Invariant	Violation example
D1	Exactly one canonical entry per title.	"Bruce Lee" appears twice.
D2	No substring-superset shadows the more-specific entry.	"Gauntlet" at line 100, "Gauntlet II" at line 600 → query for "Gauntlet II" returns the wrong row.
D3	No title-prefix that swallows unrelated titles.	DB entry "Pool" would match "Pool of Radiance", "Pool Shark", "Pool Champion" — too greedy. Minimum 6 chars or distinctive token.
D4	Canonical entry is the latest curated revision.	Old entry at line 38 with year=0, new curated entry at line 700 with full metadata → keep the line-700 entry, delete the line-38 stub.
D5	Sorted alphabetically within platform section.	New entry appended at file-end produces incorrect first-match behaviour for ambiguous queries.

D1+D2 together are what got violated in v4.1.5: 9 stale stubs at the top shadowed canonical entries below, each fix unmasked the next bug (whack-a-mole). See git show c0ea296.

Workflow

Step 1: Search before adding

# Case-insensitive search for the title core (NOT the full string —
# substring matching means even a partial collision matters)
grep -in "bruce lee" src/protection/c64/c64_protection_db.c

# Also search reverse: shorter titles that would shadow yours
grep -in '"[A-Z][a-z]\{2,8\}"' src/protection/c64/c64_protection_db.c | \
    awk -F'"' '{print $2}' | sort -u

If you get a hit: stop adding, decide which is canonical, delete the other. Never have two.

Step 2: Choose canonical fields

Required: title, protection_type, year, publisher. Optional: notes. If a field is unknown, leave the existing default — don't fabricate ("circa 1985" is fabrication; year=0 is honest unknown).

Step 3: Insert at correct alphabetical position

/* WRONG — append to end, breaks D5 + risks shadowing */
{ "Zynaps",       PROT_RAPIDLOK,    1987, "Hewson",   "..." },
{ "Bruce Lee",    PROT_VMAX,        1984, "Datasoft", "..." },  /* NEW, end */

/* RIGHT — insert alphabetically, run de-dup check */
{ "Boulder Dash", PROT_NONE,        1984, "First Star", "" },
{ "Bruce Lee",    PROT_VMAX,        1984, "Datasoft", "..." },  /* NEW */
{ "Buggy Boy",    PROT_NONE,        1987, "Elite",    "" },

Step 4: Run the lookup probe

# Build a tiny C probe (5 LOC) that calls c64_lookup_title for every
# title in the DB and asserts entry == the inserted record. Catches
# shadow bugs immediately.
gcc -I include/ tests/protection_db_selfcheck.c \
    src/protection/c64/c64_protection_db.c -o /tmp/dbcheck && /tmp/dbcheck

# Expected: "1032/1032 self-lookups returned canonical row"
# Failure mode: "Entry N at line L returns row M (shadowed)"

If a self-lookup returns the wrong row, D1 or D2 is violated.

Step 5: Run downstream tests

ctest -R "c64_protection|protection_db" --output-on-failure

test_c64_protection is the canary. If it goes red after your edit, you either inserted a shadow or perturbed an existing one. Don't relax the test — fix the data.

Pre-Done Checklist (vor dem „fertig"-Report verpflichtend)

Bevor du eine DB-Änderung als done meldest, gehst du diese Checkliste durch. Jeder Punkt = explizite Antwort. D1-D5 sind die Invarianten; die Checkliste ist der aktive Pass der sie durchsetzt.

#	Frage	Wie geprüft
P1	Substring-Suche vor Insert gemacht? Habe ich case-insensitive nach dem Title-Kern UND nach möglichen Shadow-Kandidaten gegrept?	grep -in "<core>" Output im Report zeigen, nicht „habe ich gemacht".
P2	Genau EINE Zeile pro Title? Bestätigt die Title-Uniqueness-Audit (§Verification 1) dass kein Duplikat besteht?	python3 -c "..." Ausgabe NONE im Report. D1.
P3	Kein Substring-Shadow? Audit (§Verification 2) meldet 0 D2-Violations? Spezifischer Title („Gauntlet II") wird nicht von kürzerem („Gauntlet") überschattet?	scripts/audit_protection_db.py Output mit 0 D2 violations. D2.
P4	Kein Prefix-Swallow? Mein neuer Title ist >= 6 Zeichen ODER ein distinktives Token, der keinen unrelated Title schluckt?	Audit-Output 0 D3 violations + manueller Sanity-Check der nachbarten Titel. D3.
P5	Canonical statt Stub? Wenn ich einen alten PROT_UNKNOWN, 0, "", ""-Stub finde der durch meinen Insert canonical wird: ist der Stub GELÖSCHT, nicht behalten „just in case"?	git diff zeigt eine ALTE Zeile gelöscht UND eine NEUE Zeile eingefügt. D4.
P6	Alphabetisch korrekt eingefügt? Position liegt zwischen den richtigen Nachbarn in der Plattform-Sektion, nicht ans Datei-Ende drangehängt?	git diff Kontext zeigt direkt vorausgehende + nachfolgende Zeile alphabetisch korrekt. D5.
P7	Keine fabrizierten Jahre? Wenn ich year nicht aus einer verifizierten Quelle weiß: steht 0, nicht „felt about 1985"?	Quelle für jedes != 0 year im Commit-Body ODER notes-Feld. DESIGN_PRINCIPLE 4.
P8	Self-Lookup-Probe gelaufen? Hat protection_db_selfcheck für jeden Eintrag (auch die nicht von mir geänderten) die canonical Row zurückgegeben?	1032/1032 self-lookups returned canonical row Output. Sonst → D1 oder D2 verletzt.
P9	Downstream-Tests grün? ctest -R "c64_protection|protection_db" ist grün NACH meiner Änderung?	--output-on-failure Output zeigen, nicht „läuft hoffentlich".
P10	Article-Variant entschieden? Bei „The X" / „X, The": habe ich für die Plattform-Sektion EINE Form gewählt und im Kommentar oberhalb der Sektion dokumentiert?	Code-Kommentar im Diff sichtbar.

Wenn auch nur EIN Punkt rot oder unbeantwortet: die Aufgabe ist offen. Ein latenter Shadow-Bug in der DB ist eine forensische Falschmeldung — „Bruce Lee → V-MAX" wenn es eigentlich die Datasoft-Variante ohne Protection ist, ist ein Verstoß gegen DESIGN_PRINCIPLE 1.

Common pitfalls (Beispiel-Anti-Patterns)

Die Pre-Done-Checkliste oben ist die aktive Pflicht; die folgenden Beispiele sind die Belegquellen dafür, warum es jeden Punkt gibt.

Adding a short distinctive token without checking prefix swallow

/* WRONG — "Pool" swallows Pool of Radiance, Pool Shark, ... */
{ "Pool", PROT_NONE, 1986, "Mastertronic", "" },

/* RIGHT — full canonical title */
{ "Pool BCM",   PROT_NONE, 1986, "Mastertronic", "" },
{ "Pool of Radiance", PROT_VMAX, 1988, "SSI", "" },

Keeping the old stub "just in case"

/* OLD stub at line 38 */
{ "Gauntlet", PROT_UNKNOWN, 0, "", "" },                       /* DELETE */

/* NEW canonical at line 540 */
{ "Gauntlet", PROT_RAPIDLOK, 1987, "US Gold", "Atari ST port..." },

D4: the canonical entry replaces the stub. Two entries with the same title = D1 violation = silent shadow. We're not building an audit history of the DB here — that's git log's job.

Title with leading article variants

/* English "The X" — store both forms */
{ "Last Ninja, The", ... },
{ "The Last Ninja",  ... },    /* duplicate? NO — substring-overlap, but
                                  not identical; depends on disk-name
                                  convention. Document choice in comment. */

When the disk-name convention isn't stable across releases (CSDb vs. GameBase64), pick one form, document why in a comment above the section.

Fabricating year from "felt about right"

/* WRONG — 1985 is plausible, but you don't actually know */
{ "Some Obscure Demo", PROT_FATBITS, 1985, "Unknown", "" },

/* RIGHT — year=0 = honest unknown, downstream report shows "year unknown" */
{ "Some Obscure Demo", PROT_FATBITS, 0, "Unknown", "" },

DESIGN_PRINCIPLE 4: "keine erfundenen Daten" — applies to DB curation too.

Verification

# 1. Title-uniqueness audit
python3 -c "
import re, collections
text = open('src/protection/c64/c64_protection_db.c').read()
titles = re.findall(r'^\s*\{\s*\"([^\"]+)\"', text, re.M)
dupes = [t for t, n in collections.Counter(titles).items() if n > 1]
print('Duplicate titles:', dupes if dupes else 'NONE')
"

# 2. Shadow audit (substring relationships)
python3 scripts/audit_protection_db.py src/protection/c64/c64_protection_db.c
# expected output: "0 D2 violations, 0 D3 violations"

# 3. Self-lookup probe
gcc -I include/ -o /tmp/dbcheck tests/protection_db_selfcheck.c \
    src/protection/c64/c64_protection_db.c
/tmp/dbcheck   # expected: all rows return self

# 4. Full protection test suite
ctest -R protection --output-on-failure

Open follow-up (v4.1.6+)

Per v4.1.5 release notes: ~60 additional duplicate-title entries remain in c64_protection_db.c. They are not currently test-blocking, but they are latent shadow-bugs. Two paths:

Option	Effort	Cost
A. Curate — walk the DB, merge each pair, keep richer entry	~4–6 h	One-time, finite
B. Fix lookup — replace bidirectional strstr with "longest exact match wins"	~30 LOC	Behaviour change, needs regression tests against existing callers

A preserves the API contract (callers depend on permissive substring matching for partial disk-name extracts). B breaks it but is structurally cleaner. Decide at v4.1.6 planning; don't mix into unrelated PRs.

Related

• src/protection/c64/c64_protection_db.c — 1032 entries, C64 platform
• src/protection/uft_protection_classify.c — cross-platform DB + classifier glue
• src/protection/uft_protection_extended.c — extended schemes (Amiga, Atari ST)
• include/uft/protection/uft_c64_protection.h — c64_lookup_title() declaration
• .claude/skills/uft-protection-scheme/ — when implementing a detector (not a DB entry). This skill is data; that one is code.
• .claude/skills/uft-ml-training-data/ — when assembling labeled training data for the classifier. The DB is a trusted source for labels, so D1–D5 here propagate into ML data quality.
• .claude/agents/stub-eliminator.md — old PROT_UNKNOWN, 0, "", "" stub entries qualify as D4 violations and are deletion candidates
• git show c0ea296 — the v4.1.5 commit that removed 9 shadow duplicates