audit-mobile-device-incidents

name

audit-mobile-device-incidents

description

Evidence-first audit of mobile-agent runs on real phones and cloud phones. Use when Codex must analyze task logs, acceptance JSONL, traces, screenshots, App launch failures, missing packages, installer residue, foreground instability, rendering corruption, MCP/device-session failures, failure co-occurrence, reproducibility, censoring, or conditional Agent success rates; produce device-level reports with exact denominators, device_id, session_id, and defensible root-cause boundaries.

Audit Mobile Device Incidents

Objective

Separate business failure, environment impact, system defect, Agent error, and proven device failure. Trace every count to raw evidence. Never use a broad “device/App problem” bucket as a hardware-failure rate.

Remain read-only unless the user explicitly requests output artifacts. Do not reset devices, clear App data, release sessions, or modify production services during an audit.

Every completed audit must produce two synchronized artifacts unless the user explicitly opts out:

1. a complete Markdown evidence report;
2. a single-file, degradable HTML presentation that summarizes the same findings.

Required inputs

Discover or obtain:

• batch roots and acceptance records;
• task traces, execution outcomes, runner logs, and screenshots;
• device name, full device_id, App/task, run order, and task_id;
• MCP session_id from trace execution context;
• task definitions and any human-reviewed outcome documents;
• reset policy between attempts: factory reset, pm clear, uninstall, stop App, Home only, or no cleanup.

Use rg/rg --files first. Treat summaries as leads, not facts. Inspect trace actions, foreground packages, screenshots, and MCP outcomes before assigning cause.

Read references/taxonomy.md before classifying incidents. Read references/report-template.md when producing the Markdown report. Read references/html-deck.md and reuse assets/device-audit-deck.html when producing the HTML presentation.

Workflow

1. Inventory the population

Create a unique attempt key such as (batch, device_id, case_id, run, task_id). Establish all applicable denominators before counting failures:

• total task attempts;
• attempts per device and per App/task;
• device-batch sessions;
• relevant App exposures;
• paired attempts on the same device/App/batch;
• attempts remaining after an earlier incident.

State whether attempts inherit installation, permissions, cache, login, page state, or installer overlays. Do not call stateful warm runs independent cold-start samples.

2. Build an attempt ledger

Record at least:

batch, device_name, device_id, session_id, case_id, run, task_id,
target_app, target_package, business_success, terminal_state,
failure_labels[], first_failure_time, foreground_chain,
package_membership_start/end, evidence_refs[], later_attempts_count

Preserve multiple labels on one task. Keep the primary responsibility separate from observed symptoms.

3. Determine business truth first

Judge success from the requested business outcome and visible evidence, not the Agent’s DONE or summary. Record false DONE and business success reported as non-DONE separately.

Only after fixing business truth, label why an unsuccessful attempt failed.

4. Apply strict cause boundaries

• Authentication, CAPTCHA, login, real-name checks, and risk control are business prerequisites, not device faults.
• Explicit network/service loading failures are connectivity or App-service availability, not hardware faults.
• A missing package is an initial condition. It becomes an execution impact only when installation, membership refresh, or launch fails to close.
• “Target App left foreground” is observable behavior. Call it a confirmed process crash only with exit reason, logcat, tombstone, ANR, or equivalent evidence.
• Installer residue across later tasks is test-isolation/system cleanup failure unless platform evidence proves a device defect.
• Permission-controller and installer Activities are not automatically App crashes.
• A black screenshot is not automatically display corruption; reconcile it with foreground package and adjacent frames.
• Agent error requires evidence of a wrong, omitted, repeated, or unsupported model action on an otherwise usable page.
• System error includes incorrect gates, identity/package mapping, stale-observation rejection, cleanup, lifecycle, planning, or completion acceptance.

5. Reconstruct MCP incidents

Run the bundled extractor as a first pass:

python3 scripts/extract_mcp_incidents.py <batch-root> [<batch-root> ...] --format markdown

Then verify each row against the original trace.

• Source session_id from trace execution context; never substitute task_id.
• Report full device_id and full session_id.
• Group RESOURCE_SESSION_MISMATCH by (device_id, session_id, normalized error signature) when the same failed session is reused.
• Treat client disconnect and screenshot timeout as separate events unless continuity evidence proves one persistent outage.
• Identify the first failing task, all affected tasks, failed MCP tools, secondary cleanup failures, recovery point, and whether later observations were impossible.
• Count both task impact and independent root events.

6. Audit foreground instability

Require a foreground transition chain. Strong evidence is:

1. the intended package enters foreground;
2. without an intentional App switch, foreground becomes launcher, market, or another package;
3. one deterministic relaunch reproduces the transition.

Exclude known permission overlays and intentional navigation. Break counts down by App × device × model. Calculate paired-run reproduction to distinguish random noise from compatibility concentration.

Do not claim Java/native crash, OOM kill, emulator detection, ROM incompatibility, or GPU failure without the corresponding diagnostic evidence. State the strongest supported inference and what remains unproven.

7. Audit package and installer state

Separate:

• absent at task start;
• download/prompt/install still progressing;
• installed membership observed;
• target foreground observed;
• installer UI left above later tasks;
• target launched then exited.

Model these as an ordered chain. Do not add each stage as an independent fault. If start membership snapshots are missing, report only the verifiable intersection, not a fabricated “missing package ∩ foreground exit” count.

8. Audit display evidence

Classify stripes, corrupted full frames, persistent App black screens, isolated black frames, wrong foreground, and screenshot transport failures separately. For each incident record device/App, adjacent screenshots, foreground facts, within-pair reproduction, later-batch reproduction, and confidence.

9. Analyze sequence and censoring

For every incident, ask:

• Did the runner continue to later tasks?
• Did the immediate next task succeed, repeat the issue, or fail differently?
• Did the same device-batch contain another incident type before or after it?
• Did a fatal session error remove the ability to observe later App behavior?

Treat post-session-loss tasks as right-censored for App-level phenomena. “No second problem recorded” does not prove the second problem was absent.

10. Calculate rates without double counting

Always label numerator and denominator:

• task impact rate = impacted attempts / all attempts;
• App-specific rate = incidents / relevant App attempts;
• device rate = impacted attempts / attempts on that device;
• root-event rate = independent events / device-batch sessions;
• paired reproduction = pairs with both failures / pairs with at least one failure;
• conditional Agent success = successes / genuinely Agent-evaluable attempts.

Use set union or inclusion-exclusion for multi-label technical impact. Never add overlapping branches directly. Keep raw business failure counts separate from strict technical-environment impact.

11. Produce the Markdown report

Include:

1. executive corrections to prior broad counts;
2. statistical units and denominators;
3. a Mermaid funnel or flow showing exclusions and overlaps;
4. per-device × per-task business failures;
5. strict technical-impact union;
6. foreground/App compatibility analysis;
7. package/install state analysis;
8. MCP root-event table with full device_id and session_id;
9. display/render evidence and reproducibility;
10. authentication/network exclusions;
11. co-occurrence, ordering, continuation, and censoring;
12. device risk profile and conditional Agent success interpretation;
13. source paths and known evidence gaps.

Do not include remediation when the user asks for attribution only.

12. Produce the degradable HTML presentation

Copy assets/device-audit-deck.html to the requested output location and replace its placeholder slides with the audited facts. Keep Markdown and HTML numbers synchronized from the same final ledger.

The HTML presentation must:

• be a single self-contained .html file with inline CSS and optional inline JavaScript;
• avoid CDN, remote fonts, remote JavaScript, remote CSS, and network-only images;
• use semantic <main> and <section class="slide"> markup;
• remain fully readable as a vertical document when JavaScript is disabled;
• use JavaScript only for optional keyboard navigation, progress, and slide activation;
• use @media print and @page so each slide prints as one 16:9 page;
• expose all chart values in visible labels or tables, not color alone;
• avoid hover-only evidence, animation-only meaning, canvas-only charts, and runtime Mermaid dependencies;
• include full device_id/session_id in the MCP evidence slide, or an appendix slide when density is high;
• link to the Markdown report and local evidence using relative paths where practical;
• show evidence gaps and confidence, not only conclusions.

Use HTML tables, CSS bars, and inline SVG with text labels for charts. If a complex chart cannot degrade cleanly, replace it with a table plus a short conclusion.

Validation checklist

Before handing off:

• Recompute all row and column totals.
• Confirm branch sums use de-duplicated sets.
• Confirm every MCP event has full device_id, session_id, first task, task impact, and event count.
• Confirm session reuse directly from trace records.
• Confirm authentication, explicit network failures, and missing-package state alone are excluded from device-fault rates.
• Confirm “crash” wording matches available process evidence.
• Confirm screenshot links and source paths exist.
• Explain any unavailable denominator or unrecoverable intersection instead of estimating it.
• Run Markdown/link checks appropriate to the workspace.
• Confirm both Markdown and HTML artifacts exist and report the same totals.
• Open or parse the HTML with JavaScript disabled and verify every slide remains readable.
• Confirm the HTML contains no external runtime dependency and print CSS defines one 16:9 page per slide.