// Guide for debugging failing example tests in the `test-examples` labeled workflow. Use this skill when investigating CI failures in the run-examples.yml workflow, when example scripts fail to run correctly, when needing to isolate specific test failures, or when analyzing workflow logs and failure patterns.
This skill should be used when the user asks to "trigger an eval", "run evaluation", "run swebench", "run gaia", "run benchmark", "compare eval runs", "compare evaluation results", "check eval regression", "compare benchmark results", "what changed in the eval", "diff eval runs", or mentions triggering, comparing, or reporting on SWE-bench, GAIA, or other benchmark evaluation results. Provides workflow for triggering evaluations on different benchmarks, finding and comparing runs, and reporting performance differences.
This skill should be used when the user asks to "release the SDK", "prepare a release", "publish a new version", "cut a release", "do a release", or mentions the SDK release checklist or release process. Guides through the full software-agent-sdk release workflow from version bump to PyPI publication, emphasizing human checkpoints.
This skill should be used when the user asks to "test a saas cross-repo feature", "deploy a feature branch to staging", "test SDK against OH Cloud branch", "e2e test a cloud workspace feature", "test secrets saas inheritance", or when changes span the SDK and OpenHands enterprise and need end-to-end validation against a staging deployment.
This skill should be used when the user asks to "rollout a feature", "complete feature release", "propagate SDK feature", "track feature support", "what's missing for feature X", or mentions checking CLI/GUI/docs/blog support for SDK features. Guides agents through the multi-repository feature release workflow from SDK to docs to marketing.
Project coding standards and style guidelines. Always follow these conventions when writing or reviewing code.
This skill helps encrypt and decrypt messages using ROT13 cipher. Use when the user asks to "encrypt" or "decrypt" a message.
| name | debug-test-examples-workflow |
| description | Guide for debugging failing example tests in the `test-examples` labeled workflow. Use this skill when investigating CI failures in the run-examples.yml workflow, when example scripts fail to run correctly, when needing to isolate specific test failures, or when analyzing workflow logs and failure patterns. |
The run-examples.yml workflow runs example scripts from examples/ directory. Triggers:
test-examples label to a PRModify tests/examples/test_examples.py to focus on specific tests:
_TARGET_DIRECTORIES = (
# EXAMPLES_ROOT / "01_standalone_sdk",
EXAMPLES_ROOT / "02_remote_agent_server", # Keep only failing directory
)
Add to _EXCLUDED_EXAMPLES with explanation:
_EXCLUDED_EXAMPLES = {
# Reason for exclusion
"examples/path/to/failing_test.py",
}
Toggle the test-examples label:
# Remove label
curl -X DELETE -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/OpenHands/software-agent-sdk/issues/${PR_NUMBER}/labels/test-examples"
# Add label
curl -X POST -H "Authorization: token $GITHUB_TOKEN" \
-H "Accept: application/vnd.github.v3+json" \
"https://api.github.com/repos/OpenHands/software-agent-sdk/issues/{PR_NUMBER}/labels" \
-d '{"labels":["test-examples"]}'
# Check status
curl -s -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/OpenHands/software-agent-sdk/actions/runs/{RUN_ID}" | jq '{status, conclusion}'
# Download logs
curl -sL -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/OpenHands/software-agent-sdk/actions/runs/{RUN_ID}/logs" -o logs.zip
unzip logs.zip -d logs
| Pattern | Cause | Solution |
|---|---|---|
| Port conflicts | Fixed ports (8010, 8011) | Run with -n 1 or use different ports |
| Container issues | Docker/Apptainer setup | Check Docker availability, image pulls |
| LLM failures | Transient API errors | Retry the test |
| Example bugs | Code errors | Check traceback |
Workflow (.github/workflows/run-examples.yml):
blacksmith-2vcpu-ubuntu-2404-n 4 (pytest-xdist: 4 parallel workers)Tests (tests/examples/test_examples.py):
_TARGET_DIRECTORIES_EXCLUDED_EXAMPLES