| name | zl-extractor |
| description | Use when a user asks to find, read, recover, inspect, or export Zalo PC chat history, message/pinned links, or requested chat attachments into a portable, organized folder. |
ZL Extractor
Use Zalo's logged-in runtime as the read/decryption boundary. Resolve the exact
conversation, read it without writing to Zalo, export an organized folder, and
report what was verified versus what remains partial.
Non-negotiables
- Process only the user's local account or an explicitly authorized scope.
- Never crack encryption, extract keys, upload chat data, or call send/delete/import APIs.
- A live
Core/Message partition is app-managed local data, not automatically an official backup.
- Discover every machine-specific path, account ID, conversation ID, and CDP port during the current run.
- Keep raw message bodies, signed URLs, tokens, and opaque attachment objects out of terminal output.
- Leave the source Zalo data unchanged. Temporary copies must include DB
-wal and -shm companions.
Runtime discovery
Use variables, never paths copied from a previous machine:
USER_HOME = os.homedir() / Path.home()
SKILL_ROOT = directory containing this SKILL.md
RUNTIME_ROOT = SKILL_ROOT/runtime
ZALO_DATA_ROOT = verified ZaloData directory for the active user
DB_ROOT = ZALO_DATA_ROOT/Database/_production
ACCOUNT_ID = discovered active account directory
ZALO_ACCOUNT_ID = verified active Zalo user/UIN used by the read-only adapter
GROUP_NAME = exact requested conversation display name
CONVERSATION_ID = runtime-resolved stable conversation ID
ZALO_APP_PATH = discovered Zalo.app bundle containing app.asar
CDP_PORT = free loopback port selected for this run
ZALO_READY_TIMEOUT_MS = bounded wait for the logged-in renderer (default 30000)
OUTPUT_ROOT = absolute output directory
TEMP_ROOT = fresh temporary directory
START_AT = optional lower message boundary (ISO-8601, YYYY-MM-DD, or epoch)
END_AT = optional inclusive upper message boundary (ISO-8601, YYYY-MM-DD, or epoch)
On macOS, verify USER_HOME/Library/Application Support/ZaloData and its
Database/_production directory first. On other systems, use the OS path API;
do not apply the macOS path literally. Search for an official export separately
under the verified media/account paths (.zdb or backup_zalo_*.zl.zip).
Speed, phase ledger, and resumability guardrails
The post-process scripts are fast; the expensive phases are authenticated
runtime reads and media retrieval. Keep one runtime session and one normalized
message snapshot per run. Keep the user's scope in a machine-readable run plan;
the AI must not infer extra phases after the run starts. Do not write a new
inline extractor for each group.
Use the bundled version-sensitive adapters under RUNTIME_ROOT. The snapshot
and delta adapters preserve URLs from Zalo's actual chat.recommended link
cards under the normalized structured_links column; system-event titles,
attachment names, thumbnails, and preview assets are excluded. The
post-processor consumes that column without duplicating URLs already present in message text. If the
required adapter is unavailable or its module contract no longer matches the
installed Zalo build, stop with BLOCKED and report that exact gap instead of
starting an unbounded ad-hoc scrape.
Create source/phase-ledger.json before the first runtime call. Every phase
must have started_at, finished_at, duration_ms, items, bytes,
retries, and status; write it after each phase and in the final report.
Use this fixed critical path:
resolve -> preflight -> messages -> pins -> media_prepare -> media_fetch -> post_process
Immediately after the phase ledger, create and validate
source/run-plan.json. The plan is the scope boundary for the whole run:
python3 "$SKILL_ROOT/scripts/phase_ledger.py" init "$OUTPUT_ROOT"
python3 "$SKILL_ROOT/scripts/run_plan.py" init "$OUTPUT_ROOT" \
--scope messages links pins
python3 "$SKILL_ROOT/scripts/run_plan.py" validate "$OUTPUT_ROOT"
Use only the requested scopes (messages, links, pins, media). pins
requires links; messages is always included. A phase marked SKIPPED in
the plan must not be called later. For a resumed run, load the existing plan;
do not regenerate it from the latest prompt.
Use the bundled helper instead of inventing a new ledger format:
python3 "$SKILL_ROOT/scripts/phase_ledger.py" record "$OUTPUT_ROOT" messages \
--items "$MESSAGE_ITEMS" --bytes "$MESSAGE_BYTES" --retries "$RETRIES" \
--duration-ms "$DURATION_MS" --status COMPLETE
python3 "$SKILL_ROOT/scripts/phase_ledger.py" finalize "$OUTPUT_ROOT" --status COMPLETE
python3 "$SKILL_ROOT/scripts/phase_ledger.py" validate "$OUTPUT_ROOT"
Record SKIPPED for pins when links/pins are outside scope, and for
media_prepare/media_fetch when the user did not request binaries. Record
PARTIAL or BLOCKED when a phase ends that way; never mark it complete by
guessing. validate is required before closeout.
Preflight once, before the full snapshot: use the bundled CDP readiness helper
to poll /json/list until the page title is exactly Zalo (default timeout is
30 seconds; override with bounded ZALO_READY_TIMEOUT_MS). A login/loading
page is not ready. Then verify the read-only adapter, active account, exact
conversation, pin adapter/end-marker, output root, and media field shape. Do
not inspect the app bundle, invent a new inline extractor, or rerun exploratory
pin calls after the full snapshot has started. If the renderer remains on the
login/loading page or any preflight check fails, stop BLOCKED instead of
spending time on a doomed run.
Use these defaults unless a measured 20-item media dry-run proves the renderer
requires a safer limit:
MESSAGE_BATCH_SIZE = 9000
MAX_MESSAGE_PAGES = 100
MEDIA_CONCURRENCY = 4 # use 1 when the renderer serializes requests
MEDIA_TIMEOUT = 30s per item
MEDIA_ATTEMPTS = 2 # initial attempt + one retry for 429/5xx/network only
MEDIA_DRY_RUN = 20 eligible items
MEDIA_PROGRESS = every 25 items or 10 seconds, whichever comes later
Checkpoint after each message page and each media item in the append-only item
ledger. Initialize it before the first page/media request:
python3 "$SKILL_ROOT/scripts/item_checkpoint.py" init "$OUTPUT_ROOT"
Record RUNNING before work and COMPLETE, PARTIAL, FAILED, or SKIPPED
after work. Every item needs a stable item_key and input SHA-256. Resume only
the latest PENDING/RUNNING/PARTIAL/FAILED items; a matching
COMPLETE item is immutable and the helper turns a duplicate completion into
a no-op. Validate it at closeout:
python3 "$SKILL_ROOT/scripts/item_checkpoint.py" list "$OUTPUT_ROOT" --resumable
python3 "$SKILL_ROOT/scripts/item_checkpoint.py" validate "$OUTPUT_ROOT"
Continue an existing export on a later day
Item checkpoints resume an interrupted run; the persistent incremental state
handles a new run against the same export folder. Initialize it once after the
first export:
python3 "$SKILL_ROOT/scripts/incremental_state.py" init "$OUTPUT_ROOT" \
--conversation-id "$CONVERSATION_ID"
On the next run, read source/incremental-state.json and use its watermark
(timestamp, message_id). Read newest runtime pages, keep only rows newer than
that tuple, and stop after a page crosses the watermark. Do not use a calendar
date alone: message IDs break ties and prevent same-second duplicates. Always
resolve and verify the exact conversation again. If the runtime cannot prove
ordered pagination, use a full snapshot instead of guessing.
All ordering paths normalize epoch seconds, epoch milliseconds, ISO timestamps,
and date-time strings before comparing. Numeric message IDs are compared
numerically, not lexically; a UI/API pin-count mismatch remains PARTIAL even
when an API end marker says there are no more rows.
The bundled delta adapter consumes the watermark, verifies the resolved
conversation ID, and writes only normalized fields to temporary files:
ZALO_CDP_PORT="$CDP_PORT" \
ZALO_ACCOUNT_ID="$ZALO_ACCOUNT_ID" \
ZALO_GROUP_NAME="$GROUP_NAME" \
INCREMENTAL_STATE_PATH="$OUTPUT_ROOT/source/incremental-state.json" \
MESSAGES_DELTA_PATH="$TEMP_ROOT/messages-delta.csv" \
MEDIA_CANDIDATES_PATH="$TEMP_ROOT/media-candidates.jsonl" \
node "$RUNTIME_ROOT/fetch_zalo_message_delta.mjs"
The delta CSV schema is allowlisted. Opaque runtime fields, unknown fields, and
signed internal media queries are rejected by the merge step. Both temporary
files must stay outside OUTPUT_ROOT and are deleted after post-processing.
Write the delta to a temporary normalized CSV, merge it by message_id, and
refresh the watermark:
python3 "$SKILL_ROOT/scripts/incremental_state.py" merge \
"$OUTPUT_ROOT" "$TEMP_ROOT/messages-delta.csv"
python3 "$SKILL_ROOT/scripts/incremental_state.py" validate "$OUTPUT_ROOT"
The merge is idempotent: an existing ID is updated at most once and rerunning
the same delta adds no duplicate. Then rerun link classification, the current
pin audit, requested media, and the readable renderer over the merged snapshot.
Pins are always checked live because an older message can become pinned later.
Build the unique media work queue before fetching, and remove policy-skipped /
not-found items before opening a network request. Associate a binary only by an
exact message/media reference or verified local path—never by nearest timestamp
or sender heuristic. A failed association is PARTIAL/review, not a guessed
link. JSONL is intentionally the first storage format: it is inspectable and
can be migrated to SQLite later without changing the item schema.
Never persist an opaque runtime snapshot or signed CDN query in the final raw
layer. Keep such data temporary, or store only normalized fields, host/status,
and a fingerprint. This preserves auditability without turning the export into
a reusable credential cache.
Workflow
1. Resolve the conversation
- Create the output root, phase ledger, run plan, and item checkpoint file, then discover the active account under
DB_ROOT and correlate it with the logged-in Zalo renderer.
- Use Zalo's group/conversation list (the tested build exposes
Gm1y) to map the exact display name to its ID and metadata. Do not identify a group from a DB filename.
- Require an independent check: opened conversation, preview text, recent sender/time, member count, or conversation-key membership. Stop if the name is ambiguous.
- A
file is not a database error means encrypted/app-managed state may be present; it does not prove that messages are missing.
The usual group partition is DB_ROOT/ACCOUNT_ID/Core/Message/g<GROUP_ID>.db,
but mapping must come from Zalo first. Official exports and live partitions are
different sources.
2. Read through the logged-in runtime
Use a temporary loopback CDP connection only when needed. Select the page titled
Zalo from /json/list. Use the app's read-only DataAccess service; module
IDs such as AY7h are version-dependent. If the tested module is absent, stop
BLOCKED; do not replace it with a new inline extractor.
For a first/full snapshot, write normalized messages directly to the export's
source/raw/ layer and keep authenticated media candidates outside the export:
ZALO_CDP_PORT="$CDP_PORT" \
ZALO_ACCOUNT_ID="$ZALO_ACCOUNT_ID" \
ZALO_GROUP_NAME="$GROUP_NAME" \
OUTPUT_ROOT="$OUTPUT_ROOT" \
START_AT="${START_AT:-}" END_AT="${END_AT:-}" \
MESSAGES_PATH="$OUTPUT_ROOT/source/raw/messages.csv" \
MEDIA_CANDIDATES_PATH="$TEMP_ROOT/media-candidates.jsonl" \
node "$RUNTIME_ROOT/fetch_zalo_message_snapshot.mjs"
The snapshot adapter initializes source/manifest.json with the verified
conversation, read-only source metadata, message counts, and
sourceWriteIssued: false; later link/pin stages extend that manifest instead
of silently omitting it.
START_AT/END_AT are inclusive. A date-only START_AT means local midnight;
a date-only END_AT means the end of that local day. Numeric epoch seconds and
milliseconds are accepted. The adapter refuses ambiguous group names, missing
active-account identity, repeated cursors, page-cap overflow, and unsafe output
paths; an invalid timestamp never becomes the stop condition and is excluded
when a date boundary is active.
Paginate with the app cursor, never SQL OFFSET:
cursor = "9999999999999"
repeat:
batch = loadMessagesForBackup(conversation_id, cursor, 9000)
append batch
stop when batch.length < 9000
cursor = msgId of the last record
Guard against a repeated cursor and cap pages. Normalize only the fields needed
for output (conversation_id, message ID, timestamps, sender, type, text,
quote/reference, and attachment metadata), then sort oldest-to-newest by
sendDttm, msgId. Message-text links are incomplete until the pin audit runs.
START_AT and END_AT apply only to the chronological message snapshot; they
never filter the independent pin panel. A pin whose original message is
outside the window must still be retained as a pin record and marked
out-of-window in provenance.
Recognize explicit http(s):// URLs and conservative bare domains; keep
ambiguous domain-like prose in the review queue instead of guessing.
Use one runtime call per page, not one CDP evaluation per message. Record the
page cursor and input hash as one checkpoint item, persist the normalized page
before requesting the next page, and never fetch a completed page again.
When media is in scope, derive candidate rows while each message page is
already in memory and write them to a temporary
TEMP_ROOT/media-candidates.jsonl file. Each row contains msgId, sendDttm,
url, urlKey, and optional normalized media metadata; the file is staging data
and must stay outside the final output root. Pass it to the media fetcher as the required
MEDIA_CANDIDATES_PATH. The fetcher may attach to CDP for cookies but must not
call loadMessagesForBackup again. A missing or invalid candidate file is
BLOCKED; do not silently fall back to a second history scan. Candidate
validation allows only known Zalo media host families; external URLs are
rejected before any CDP cookie lookup.
Media downloads stream into a temporary file, compute SHA-256 while receiving,
validate the magic bytes/MIME, then rename into the final attachment folder.
Existing files are hash-checked before reuse. This keeps large videos/files out
of RAM and leaves no partial final binary after an interrupted download.
3. Process links and pinned content
When links are in scope, read references/links-and-pins.md
before extraction. It defines the exact pin audit, URL dedupe boundary,
context merge, classification rules, review queue, and link output schema.
Open the exact conversation and its visible pin panel first, then audit it with
the bundled read-only adapter. The visible panel is used only as a count
evidence check; the adapter still reads the bounded pin service. It writes a
temporary normalized pin table and a small audit record; move only the table
into source/raw/:
ZALO_CDP_PORT="$CDP_PORT" \
ZALO_GROUP_NAME="$GROUP_NAME" \
OUTPUT_ROOT="$OUTPUT_ROOT" \
PINS_PATH="$TEMP_ROOT/pins.csv" \
PIN_AUDIT_PATH="$OUTPUT_ROOT/source/pin-audit.json" \
node "$RUNTIME_ROOT/fetch_zalo_pins.mjs"
mv "$TEMP_ROOT/pins.csv" "$OUTPUT_ROOT/source/raw/pins.csv"
If the app's pin service cannot resolve the exact conversation or does not
return an array from the bounded read-only call, record BLOCKED/PARTIAL and
do not claim pin completeness. An API-reported row count alone is not an end
signal; require an exact UI total or an explicit no-more/end marker. When a
visible pin panel shows older records than the message window, hydrate the pin
record independently and keep it even when its original message is outside
START_AT.
If the exact pin panel is not visible or its count cannot be bound to the
requested conversation, keep the audit PARTIAL; do not treat the API row
count as a substitute.
The conversation-info Link tab is a separate source from both the message
snapshot and the pinned-message panel. Open the exact conversation's full
Link view, then use the bundled adapter:
ZALO_CDP_PORT="$CDP_PORT" \
ZALO_GROUP_NAME="$GROUP_NAME" \
OUTPUT_ROOT="$OUTPUT_ROOT" \
LINK_ARCHIVE_PATH="$OUTPUT_ROOT/source/raw/link-archive.csv" \
LINK_ARCHIVE_AUDIT_PATH="$OUTPUT_ROOT/source/link-archive-audit.json" \
ZALO_REPORTED_LINK_COUNT="${ZALO_REPORTED_LINK_COUNT:-}" \
node "$RUNTIME_ROOT/fetch_zalo_link_archive.mjs"
The Zalo number is a card count, not a URL count: one card may contain many
exact URLs. The adapter scrolls to a stable end, writes one row per card, and
records reportedCardCount, enumeratedCardCount, and the end condition. The
link extractor reads URLs from the card title and uses href only when the
title has no URL; preview images/assets are never treated as shared links. If it is unavailable,
keep the export PARTIAL and show that gap in readable/index.md; never use
the message/pin totals as a substitute for the Link-tab count.
Build the occurrence ledger and exact-URL merge before applying category rules:
python3 -B scripts/extract_links.py "$OUTPUT_ROOT"
python3 -B scripts/apply_category_rules.py "$OUTPUT_ROOT"
Run deterministic rules first. Send only uncertain/conflicting rows plus their
related occurrences to the review queue; do not send the whole conversation or
one AI call per URL.
Run these static smoke tests once after changing the skill or renderer:
python3 -B scripts/test_link_rules.py
python3 -B scripts/test_human_views.py
python3 -B scripts/test_incremental_state.py
python3 -B scripts/test_time_order.py
python3 -B scripts/test_extract_links.py
node runtime/test_browser_runtime.mjs
node runtime/test_message_order.mjs
node runtime/test_link_archive_contracts.mjs
node runtime/test_full_snapshot_contracts.mjs
node runtime/test_pin_contracts.mjs
node runtime/test_media_contracts.mjs
node runtime/test_media_stream.mjs
node runtime/test_zalo_cdp_contracts.mjs
python3 -B scripts/test_stress_pipeline.py --messages 4000 --pins 800
python3 -B scripts/test_stress_pipeline.py --messages 12000 --pins 2000
For each export, run only the data-dependent pipeline after the runtime
snapshot exists:
python3 -B scripts/extract_links.py <OUTPUT_ROOT>/<slug>-export-<timestamp>
python3 -B scripts/apply_category_rules.py <OUTPUT_ROOT>/<slug>-export-<timestamp>
python3 -B scripts/write_link_review.py <OUTPUT_ROOT>/<slug>-export-<timestamp>
python3 -B scripts/enforce_attachment_policy.py <OUTPUT_ROOT>/<slug>-export-<timestamp>
python3 -B scripts/render_human_views.py <OUTPUT_ROOT>/<slug>-export-<timestamp>
python3 -B scripts/audit_links.py <OUTPUT_ROOT>/<slug>-export-<timestamp>
When the user supplies a manual URL list, reconcile exact URL coverage and
keep line-level occurrence evidence instead of comparing totals by eye:
python3 -B scripts/reconcile_link_baseline.py "$OUTPUT_ROOT" "$MANUAL_LINKS_FILE"
Repeated entries are retained as OCCURRENCE_NOT_PROVEN when Zalo does not
independently show a matching second occurrence; they do not turn a present
exact URL into a false missing-link claim. write_link_review.py writes review files. audit_links.py is read-only and
returns 0=PASS, 2=PARTIAL, 1=FAIL. It checks exact per-URL occurrence
counts, not only partition totals, and rejects signed internal media queries.
4. Retrieve attachments only when requested
Read references/attachments.md before copying or
downloading binaries. GIFs and stickers are excluded by default and recorded as
skipped_by_policy with no output path; do not silently omit other requested
media.
5. Write the export
Read references/readable-output.md before
writing a folder intended for people. Write outside the source DB, under a
resolved absolute path, with the readable layer first:
<OUTPUT_ROOT>/<slug>-export-<timestamp>/
readable/ Markdown reading views + curated CSV tables
source/
attachments/ only requested and verified binaries
raw/ machine-readable audit inputs; signed media queries redacted
manifest.json, run plan, checkpoints, provenance, phase timings
Generate the human views after the normalized message/link/media files exist:
python3 scripts/render_human_views.py <OUTPUT_ROOT>/<slug>-export-<timestamp>
readable/messages.md is chronological and grouped by local date. links.csv
is the single compact, one-row-per-canonical-URL reader table with URL,
category, occurrence count, first-send time, and review status. The renderer
orders links.csv by normalized platform/domain/brand family first, then by
first-send time within that family; category is only a tie-breaker. Equivalent
vmedia.vn/vmedia.ai, YouTube/youtu.be, Facebook, TikTok, and Telegram
variants stay adjacent. It then renumbers the displayed STT. The original
sequence stays in source/raw/links.csv. pins.md is
the separate reader-facing pin audit; external links and internal media
references are shown separately and must not be hidden inside the general link
count. media.csv and review.csv are created only when they contain rows.
Long context and source evidence stay in source/raw/ and the optional review table.
Keep confidence, IDs, classification evidence, hashes, and other provenance in
the source/raw/ layer; never make a reader open raw CSV to understand the conversation.
Do not create XLSX unless formulas, pivots, or a styled workbook are explicitly
requested. The renderer escapes untrusted message/context Markdown, protects
formula-like readable CSV cells, preserves external URLs exactly, and redacts
internal signed Zalo media URLs. It is non-destructive and can mirror a legacy
01-messages/ + 03-reports/ export into the new layers.
Use UTF-8 and real CSV quoting. Use collision-safe filenames and relative paths
in CSV/manifest. Never overwrite existing binaries or edit immutable source
inputs; derived ledgers may be regenerated only after the source-write guard is
checked. Rerun the renderer and audit after moving anything.
6. Validate and report
Read references/verification.md before closeout.
Validation must come from the raw ledger/source messages, not only the
classifier's report. Verify conversation mapping, counts, unique message IDs,
CSV round-trip, dedupe/media partitions, readable/raw reconciliation,
path containment, hashes, and pin coverage. Keep the source-write flag false.
Require run_plan.py validate, item_checkpoint.py validate, and
phase_ledger.py validate to pass. If timings, retries, plan, or checkpoints
are missing, report PARTIAL with the specific missing state rather than
claiming the workflow was fast or complete.
Use these statuses:
COMPLETE = mapping, requested records, links/pins, and requested media are verified
PARTIAL = records are readable but pin coverage, classifications, or requested media remain unresolved
BLOCKED = no valid source/runtime, ambiguous conversation, or inconsistent source
Never call a text export complete when the requested pin audit is unknown or
blocked. An expected GIF/sticker exclusion does not make an otherwise complete
in-scope export partial.
7. Clean up
Remove temporary scripts, extracted app bundles, cloned repositories, and
temporary package directories. Close the temporary CDP session and relaunch
Zalo normally if it was started with a debug port. Never delete the original
Zalo data without a separately approved, recoverable plan.
