data-mismatch-debugging

name	data-mismatch-debugging
description	Debugging data mismatch issues — when files exist but references are broken. Checklist for avoiding common pitfalls during debugging.
tags	["debugging","data-integrity","best-practices","ops"]

Data Mismatch Debugging Checklist

When "something works on machine A but not machine B", the root cause is often data mismatch, not infrastructure.

Debugging Order (MUST follow)

1. DATA LAYER     — Do referenced files exist? Do IDs match?
2. APPLICATION    — Are APIs returning correct content?
3. INFRASTRUCTURE — Network, DNS, proxy, tunnel?

Why this order: Infrastructure issues affect everything. Data issues affect specific items. If 1 out of N items works, it's almost always a data problem, not infrastructure.

The 5 Anti-Patterns

1. Deleting production data during debugging

❌  rm -rf /data/important_dir/   # "I'll regenerate it"
✅  cp -r /data/important_dir/ /data/important_dir.bak/

Rule: Never delete without backup. Never assume regeneration produces identical output.

2. Blindly changing configuration

❌  Disabling features without understanding dependencies
✅  Read the docs, understand the dependency chain, then change

Rule: If you don't know what a config does, don't change it.

3. Starting from infrastructure instead of data

❌  "Let me check the tunnel, DNS, CORS, proxy..."
✅  "Let me check if the file exists and the ID matches"

Rule: If N items exist but only M work (M < N), the problem is per-item data, not system-wide infrastructure.

4. Using APIs without understanding side effects

❌  Calling regenerate/rebuild APIs without knowing they create new IDs
✅  Read the API docs, test on one item, verify the result

Rule: Any API that creates/renames files may change IDs. Old references become invalid.

5. Trusting cached state as ground truth

❌  "It works on my machine, so the data is fine"
✅  "It works on my machine — is that cached or live?"

Rule: Clear cache, use incognito, test from a fresh device before concluding "it works".

Quick Diagnostic Commands

# Check if referenced file exists
ls -la /path/to/referenced/file

# Check if ID in JSON matches file on disk
python3 -c "import json; d=json.load(open('data.json')); print(d['ref_id'])"
ls /path/to/files/ | grep $(python3 -c "...")

# Check from external perspective (not localhost)
curl -I https://external-url/path/to/resource

# Clear cache test
# Chrome: F12 → Application → Storage → Clear site data
# Or use incognito mode

When to Apply

• "Works on my machine but not on other devices"
• "Some items work, others don't"
• "It worked before, now it doesn't" (after a migration/sync)
• "API returns 200 but content is wrong"
• "Regenerated data but old references still point to old IDs"

Case Studies

• references/openmaic-audio-case-study.md — 717 classrooms had broken audio references after debugging deleted and regenerated files with new random IDs

Real-World Incident: OpenMAIC Audio (2026-06-27)

Timeline:

1. Audio worked on Mac browser but not on mobile/HP
2. Assumed browser autoplay policy → created /enable-audio page → didn't help
3. Checked audio URLs → curl returned 200 → "must be fine"
4. Checked IndexedDB → 0 audio files → "not a cache issue"
5. Finally checked if audioId in JSON matches actual filename on disk → MISMATCH

Root cause: Earlier debugging session deleted audio/ directory, then called regenerate-tts API which generated NEW files with NEW random IDs. JSON still referenced OLD IDs.

Fix applied:

# Match audio files to speech actions by index (not by name)
all_speech_actions = []  # from all scenes
all_audio = sorted(os.listdir(audio_dir))  # actual files on disk

for i, action in enumerate(all_speech_actions):
    if i < len(all_audio):
        audio_file = all_audio[i]
        action["audioId"] = audio_file.replace(".mp3", "")
        action["audioUrl"] = f"https://domain/api/classroom-media/{id}/audio/{audio_file}"

Key insight: Mac browser worked because it had HTTP-cached the old audio URLs. Other devices had no cache → hit the server → 404.

Lesson: "Works on my machine" often means "cached on my machine". See references/openmaic-audio-case.md for full case details.

OpenMAIC Audio Mismatch — Full Case Study

Context: OpenMAIC classrooms on HP server had audio files on disk but 404 when accessed via URL.

Root cause chain:

1. Original data synced from Mac → JSON and audio files matched ✅
2. Debugging session deleted audio/ directory ❌
3. Called regenerate-tts API → generated NEW files with NEW random IDs
4. JSON still referenced OLD IDs → 404

Fix script (batch repair all classrooms):

import os, json, re
base = "/opt/openmaic/data/classrooms"
for dirname in sorted(os.listdir(base)):
    dirpath = os.path.join(base, dirname)
    json_path = os.path.join(dirpath, "classroom.json")
    audio_dir = os.path.join(dirpath, "audio")
    if not os.path.isfile(json_path) or not os.path.isdir(audio_dir):
        continue
    with open(json_path) as f:
        classroom = json.load(f)
    existing_audio = sorted(os.listdir(audio_dir))
    classroom_id = dirname.split("_")[0]
    all_speech = []
    for scene in classroom.get("scenes", []):
        for action in scene.get("actions", []):
            if action.get("type") == "speech":
                all_speech.append(action)
    audio_by_scene = {}
    for f in existing_audio:
        m = re.match(r"tts_s(\d+)_action_(.+)\.mp3", f)
        if m:
            audio_by_scene.setdefault(int(m.group(1)), []).append(f)
    all_audio = []
    for sn in sorted(audio_by_scene.keys()):
        all_audio.extend(sorted(audio_by_scene[sn]))
    for i, action in enumerate(all_speech):
        if i < len(all_audio):
            af = all_audio[i]
            action[] = af.replace(, )
            action[] = 
     (json_path, )  f:
        json.dump(classroom, f, indent=, ensure_ascii=)

Key insight: Scene numbering in JSON may not match audio file naming. Always match by index order, not by scene number.