This skill should be used when analyzing failed GitHub Actions CI/CD runs for Breenix kernel development. Use for diagnosing test failures, parsing QEMU logs, identifying kernel panics or faults, understanding timeout issues, and determining root causes of CI failures.
This skill should be used when analyzing failed GitHub Actions CI/CD runs for Breenix kernel development. Use for diagnosing test failures, parsing QEMU logs, identifying kernel panics or faults, understanding timeout issues, and determining root causes of CI failures.
CI Failure Analysis for Breenix
Systematically analyze and diagnose CI/CD test failures in Breenix kernel development.
Purpose
This skill provides tools and workflows for analyzing failed CI runs, understanding kernel crashes, identifying environment issues, and determining root causes. It focuses on the unique challenges of kernel development CI: QEMU logs, kernel panics, double faults, page faults, and timeout analysis.
When to Use This Skill
Use this skill when:
CI run fails: GitHub Actions workflow fails and you need to understand why
Test timeout: Test exceeds time limit and you need to determine if it's a hang or just slow
Kernel panic/fault: Double fault, page fault, or other kernel crash in CI
Timeout reached (60s)
... OR ...
Error: test exceeded time limit
Distinguishing Hang vs Slow:
Kernel hang: No new output for extended period
Timer interrupt not firing
Infinite loop
Deadlock
Legitimately slow: Continuous output, just takes longer
CI environment slower than local
Verbose logging enabled
Many tests in sequence
Diagnosis:
Check last log message - what was kernel doing?
Is timer interrupt still firing? (look for timer ticks)
Are there any locks being acquired?
Does it complete locally?
Fixes:
Infinite loop: Add timeout or fix logic
Deadlock: Review lock acquisition order
Slow test: Increase timeout or optimize
Hang: Add debug checkpoints to narrow down location
Missing Success Signal
Symptoms:
❌ Ring-3 smoke test failed: no evidence of userspace execution
Common Causes:
Test didn't run (compilation failed silently)
Kernel panicked before reaching test
Test ran but failed assertions
Signal string changed but test wasn't updated
Diagnosis:
Search log for ANY output from the test
Check if kernel reached test execution point
Look for earlier errors or panics
Verify signal string matches test code
Compilation Error
Symptoms:
error[E0...]: ...
--> kernel/src/...
Common Causes:
Wrong Rust nightly version
Missing features
Syntax error
Dependency version mismatch
Diagnosis:
Check Rust version in CI vs. expected
Verify all required crates are available
Look for changed dependencies
Check for feature flag mismatches
Environment Issues
Symptoms:
qemu-system-x86_64: command not found
... OR ...
error: target 'x86_64-unknown-none' may not be installed
Common Causes:
System dependencies not installed
Rust components missing
Wrong Rust installation method
PATH not set correctly
Diagnosis:
Check workflow YAML for dependency installation
Verify Rust toolchain setup
Check for typos in package names
Confirm correct ubuntu version
Analysis Workflow
Step 1: Identify Failure Type
Download artifacts from failed GitHub Actions run
Check Actions summary for which step failed
Determine failure category:
Build failure (compilation)
Environment setup failure (missing deps)
Test execution failure (kernel crash, timeout, wrong output)
Step 2: Automated Analysis
# Run the analyzer on downloaded logs
ci-failure-analysis/scripts/analyze_ci_failure.py \
--context \
target/xtask_*_output.txt
Review the output for:
Detected patterns
Suggested diagnosis
Recommended fixes
Step 3: Manual Analysis
If automated analysis doesn't find clear patterns:
# Search for specific error keywords
grep -i "error\|panic\|fault\|timeout" target/xtask_*_output.txt
# Find last successful operation
grep "SUCCESS\|✓\|✅" target/xtask_*_output.txt | tail -20
# Look for specific subsystem activity
grep "memory\|page table\|process\|syscall" target/xtask_*_output.txt
Step 4: Reproduce Locally
# Run exact same command as CI
cargo run -p xtask -- ring3-smoke
# Or use quick debug for faster iteration
kernel-debug-loop/scripts/quick_debug.py --signal "EXPECTED_SIGNAL" --timeout 30
Step 5: Compare Environments
Aspect
Local
CI
Rust version
Check with rustc --version
Check workflow YAML
QEMU version
qemu-system-x86_64 --version
ubuntu-latest package
Timeout
Usually 30s
Usually 60s
Build cache
Warm
Cold or partial
System load
Low
Variable
Step 6: Root Cause Analysis
Document findings using the systematic debugging pattern:
Problem: What failed?
Root Cause: Why did it fail?
Solution: What fixes it?
Evidence: How do you know it's fixed?
Integration with Other Skills
Use with kernel-debug-loop
After identifying a failure, use kernel-debug-loop for rapid iteration:
# Test fix with quick feedback
kernel-debug-loop/scripts/quick_debug.py \
--signal "🎯 KERNEL_POST_TESTS_COMPLETE 🎯" \
--timeout 15
Use with github-workflow-authoring
Fix workflow issues:
# If environment issue detected:# 1. Identify missing dependency from analyzer output# 2. Update workflow using github-workflow-authoring skill# 3. Test change in PR
Use with systematic-debugging
Document the failure:
# Problem
CI run #123 failed with page fault at 0x10001082
# Root Cause
[Fill in after analysis]
# Solution
[Fill in after fix]
# Evidence
[Fill in after verification]
Advanced Techniques
Diff Analysis
Compare working vs broken runs:
# Download logs from last successful run and failed run
diff -u successful_run.txt failed_run.txt | less
Look for:
First point where outputs diverge
Missing initialization steps
Different memory addresses (ASLR not implemented, so addresses should match)
Document patterns: Add new patterns to analyzer when discovered
Test fixes: Verify fix locally before pushing to CI
Example Analysis Session
# 1. Download artifact from failed CI run# Save to: target/xtask_ring3_smoke_output.txt# 2. Run automated analysis
ci-failure-analysis/scripts/analyze_ci_failure.py \
--context target/xtask_ring3_smoke_output.txt
# Output shows: Page Fault at 0x10001082# 3. Search for context
grep -B10 -A10 "0x10001082" target/xtask_ring3_smoke_output.txt
# 4. Identify: copy_from_user failing# 5. Check if this address is mapped
grep "process page table\|mapping" target/xtask_ring3_smoke_output.txt
# 6. Hypothesis: User buffer not mapped in process page table# 7. Review recent changes to process memory code# 8. Identify fix needed# 9. Test locally with quick iteration
kernel-debug-loop/scripts/quick_debug.py \
--signal "USERSPACE OUTPUT" \
--timeout 10
# 10. Verify fix works# 11. Push to PR, monitor CI
Summary
CI failure analysis for Breenix requires:
Automated pattern detection for common failures
Manual log analysis for novel issues
Environment comparison (local vs CI)
Systematic root cause investigation
Integration with debugging and testing workflows
Documentation of findings
The analyze_ci_failure.py script automates common pattern detection, but kernel debugging ultimately requires understanding the code, memory management, interrupt handling, and the specific feature being tested.
