github-action-failure-investigation

name	github-action-failure-investigation
description	Guidelines for fetching, downloading, and analyzing failed GitHub Actions runs, including raw logs, packet captures, and detecting runner freezes.

GitHub Action Failure Investigation

This skill provides step-by-step guidelines to retrieve and analyze failed GitHub Actions runs, check job logs and packet capture artifacts, and classify failure causes (real bugs vs. racy/frozen runners).

1. Prerequisites

Make sure you can run the GitHub CLI (gh) and are authenticated:

gh auth status

Note: If running inside a sandboxed environment that injects dummy tokens (causing HTTP 401 errors), you may need to prefix commands with env -u GITHUB_TOKEN to force gh to fall back to your normal credentials.

2. Fetching Job Metadata and Logs

Always create a dedicated staging directory to store logs and artifacts to avoid cluttering the out/ root or dirtying the code source tree:

mkdir -p out/gha/<run_id>/

List Run Status and Artifacts

Use the Run ID from the URL (e.g., https://github.com/project-chip/connectedhomeip/actions/runs/<run_id>) to check status and see available artifacts:

gh run view <run_id>

Download Raw Job Logs

If the run has completed, download the logs for a specific job:

gh run view <run_id> --job <job_id> --log > out/gha/<run_id>/job_logs.log

If the run is still in progress (but the specific job has completed), download the logs via the GitHub REST API directly:

gh api repos/project-chip/connectedhomeip/actions/jobs/<job_id>/logs > out/gha/<run_id>/job_logs.log

---

3. Downloading Capture Files (PCAPs)

If the job summary indicates packet captures were uploaded (e.g., pcaps-linux-repl-TC_[A-C]), download them to the staging folder:

gh run download <run_id> -n <artifact_name> --dir out/gha/<run_id>/

---

4. Investigating Log Failures

1. Find Failed Tests: Search or grep for test summary lines to quickly identify which test classes failed or had errors:
   • Query: Summary for test class
2. Examine Tracebacks: Search or grep for tracebacks or error blocks around the failed test case:
   • Query: Traceback (most recent call last) or FAIL or ERROR
3. Inspect Logs: Open the log file to inspect the context (approx. 100-200 lines) surrounding the failure line.

---

5. Classifying the Root Cause

A. Real Issues / Bugs

• Look for C++ assertions, division by zero, null pointer access, or Python test assertions (e.g., AssertionError: True is not False).
• Trace back to the source code to locate the logic flaw.

B. Runner Freeze / Time Compression (Racy/Timing Issues)

GitHub Actions runners (especially ASAN/TSAN runs under heavy load) can freeze or experience CPU starvation. When the virtual machine is starved and then gets a burst of CPU, the virtual timers catch up in a rapid burst upon VM resumption (tick catch-up).

To detect this, compare the host runner timestamp (prefix of each log line) with the guest application log timestamp (usually in brackets):

Host Runner Timestamp: 2026-06-18T12:18:11.1232136Z [2026-06-18 12:17:58.164368] ...
Host Runner Timestamp: 2026-06-18T12:18:11.1751074Z [2026-06-18 12:18:01.925009] ...

• General Analysis:

  • Compare the host elapsed time and guest elapsed time over a sequence of log events.
  • If a significant amount of guest time (e.g., seconds) passes in a tiny fraction of host wall-clock time (e.g., milliseconds), virtual time compression is occurring due to CPU starvation/VM freeze.

• General Impact of Time Compression:

  • Any time-based logic, timers, delays, or backoff protocols (such as timeouts, retries, polling loops) will expire/execute instantly or in rapid succession when time is compressed.
  • This leads to false failures, timeouts, session drops, or retry exhaustions because the asynchronous event loop doesn't get enough wall-clock CPU time to process standard socket queues or network buffers before the retry limit is hit.

• Example (Case Study: MRP Timeout): In the example timestamps above:

  • Host elapsed time: 52 milliseconds.
  • Guest elapsed time: 3.761 seconds.
  • Result: The Message Reliability Protocol (MRP) backoff timers expired instantly one after another. All retry attempts (4 retries over ~4 seconds of virtual time) were sent and exhausted within 52ms of host time, causing premature session timeouts before the receiving event loop got CPU time to process the socket queue.

---

6. Analyzing Packet Capture (PCAP)

Use tcpdump to print loopback or network bridge interface packets:

tcpdump -nn -r out/gha/<run_id>/<pcap_file>

Verify the following:

• Were UDP packets actually transmitted on loopback (lo In / lo Out)?
• Did the receiver reply with StandaloneAck?
• If the packets are present in the capture but the receiver logs show no Msg RX events, the receiver was either frozen, starved of CPU, or its virtual time was compressed, preventing it from reading the socket buffer before the timeout occurred.