automated-smoke-test

name	automated-smoke-test
description	Run an automated smoke test using the godot-mcp server. Launches the project, captures debug output, and checks for errors or crashes.
argument-hint	[duration-seconds]
user-invocable	true
allowed-tools	Read, Glob, Grep, Write, Bash, Task, question

Automated Smoke Test

This skill runs a fully automated smoke test against the Godot project using the godot-mcp server. It launches the project headlessly, captures debug output for a configurable duration, and analyzes the output for errors, warnings, and assertions — producing a structured pass/fail report.

No manual verification required. The entire check is automated through MCP.

---

Phase 1: Verify godot-mcp Availability

Call get_godot_version via the godot-mcp server. If the call succeeds, note the version string. If it fails, inform the user:

"godot-mcp server is not available. Install it with: npx @coding-solo/godot-mcp Then configure the MCP server in opencode.json (OpenCode) or pi.json (Pi)."

Stop if the server is unavailable.

---

Phase 2: Read Project Info

Call get_project_info via godot-mcp. Record:

• Project title — from the project info response
• Main scene — the configured main scene path
• Render mode — e.g. Forward+, Mobile, GL Compatible

If get_project_info fails, retry up to 3 times with a 2-second delay between attempts. If all retries fail: "Could not read project info after 3 attempts. Is the godot-mcp server running against the correct project?" Then stop.

---

Phase 3: Run the Project

Parse the optional argument for duration. If no argument is provided, default to 10 seconds. The argument is in seconds: /automated-smoke-test 15 means capture output for 15 seconds.

Call run_project via godot-mcp. The project must respond (start or error) within 30 seconds. If no response within the timeout, treat as a failure:

• Report: "Project failed to respond within 30 seconds — the project may be hung or the engine may have frozen during launch."
• Verdict: FAIL
• Skip to Phase 7 (Report) — the project is unresponsive, stop would also hang

If run_project returns an error or the project fails to start:

• Report: "Project failed to start with error: [error message]"
• Verdict: FAIL
• Skip to Phase 7 (Report) — project was never launched, no stop needed

---

Phase 4: Capture Debug Output

Duration scaling: The capture duration should reflect project complexity. A minimal 2D project may produce output in 5 seconds; a large 3D project with many scenes may need 30+ seconds. Default to 10 seconds but consider the project's scope (from Phase 2's project info) and scale up for complex titles. For headless CI runs, prefer longer durations to account for slower hardware.

If get_debug_output returns an error:

• Report: "Could not capture debug output: [error message]"
• Verdict: FAIL
• Skip to Phase 6 (stop project), then continue to Phase 7 for the report

Do not use a fixed sleep. Instead, poll get_debug_output in a loop:

1. Every 2 seconds, call get_debug_output.
2. If the output contains any ERROR, crash, or assertion pattern (see Phase 5), stop polling early — the test has already found failures.
3. If no errors appear, continue polling until the configured duration elapses (default: 10 seconds, configurable via argument).
4. If get_debug_output returns an error on any poll tick:
   • Report: "Could not capture debug output on poll attempt [N]: [error message]"
   • Continue polling (do not abort) unless 3 consecutive polls fail.
   • After 3 consecutive failures: "Debug output capture failed after 3 consecutive poll errors."
   • Verdict: FAIL
   • Skip to Phase 6

Once polling ends (duration elapsed or early-stop triggered), use the last successful output for analysis. If all polls failed (3 consecutive errors), there is no output to analyze — skip directly to the FAIL verdict.

If the final output is empty or trivially short, note: "Output appears minimal — the project may not have rendered any frames."

---

Phase 5: Analyze Output

Scan the debug output for:

Pattern	Severity	Flags
ERROR	Error	Catch-all for Godot error messages
error:	Error	Lower-case variant in scripts
crash	Critical	Game crashed during runtime
NullReferenceException	Error	Null access in C# script (.NET)
segfault	Critical	Memory access violation
segmentation fault	Critical	Full-form segfault message
WARNING	Warning	Non-fatal warnings
warning:	Warning	Lower-case variant
Assertion failed	Error	GDScript or C# assertion failure

Count the occurrences of each pattern. Record the actual matching lines (up to 10 per pattern for the report).

---

Phase 6: Stop the Project

Call stop_project via godot-mcp to clean up. If it fails, note: "Could not stop the project cleanly — you may need to close the Godot editor or kill the process manually."

---

Phase 7: Report Results

Format the report:

## Automated Smoke Test Report

**Date**: [date]
**Project**: [project title]
**Main Scene**: [main scene path]
**Godot Version**: [version from Phase 1]
**Duration**: [X seconds]

---

### Results

| Check | Result |
|-------|--------|
| Project launched | ✅ / ❌ |
| No runtime errors | ✅ / ❌ (N errors found) |
| No critical crashes | ✅ / ❌ (N crashes detected) |
| No warnings | ✅ / ⚠️ (N warnings) |
| No assertion failures | ✅ / ❌ (N assertions failed) |

---

### Error Details

[If errors/crashes found, include the matching lines in a code block.
Otherwise: "No errors detected."]

---

### Warning Details

[If warnings found, include the matching lines in a code block.
Otherwise: "No warnings detected."]

---

### Verdict: [PASS | FAIL | SILENT-FAIL]

**FAIL** if ANY of:
- Project failed to start
- Runtime errors or crashes detected
- Assertion failures found
- Debug output could not be captured after retries

**SILENT-FAIL** if:
- Project launched successfully AND no errors/crashes detected BUT
  debug output was empty or trivially short (zero or near-zero lines).
  This means the project may have started but produced no frames or
  lifecycle output — a configuration problem or silent hang. The user
  should verify manually.

**PASS** if ALL of:
- Project launched successfully
- Debug output contains substantive content (not SILENT-FAIL threshold)
- No runtime errors or crashes
- No assertion failures
- Warnings are acceptable (advisory only — do not cause FAIL)

Present the report to the user. Do not write it to a file unless asked.