| name | runtime-evidence |
| description | Use when an evaluator needs to interact with a running application before scoring dimensions |
Runtime Evidence
Runtime evidence means interacting with the running application, not just reading
source code. Observations of actual behavior in a live environment — screenshots,
HTTP responses, console output, interaction results — are runtime evidence. Static
code analysis is not.
When a runtime is configured for this evaluation, exercise the running app and
record what you observed before scoring any dimension. Source code shows what the
implementation intends; only the live app shows what it does, so a runtime
dimension scored from code alone is scored on nothing.
Evidence Gathering Methods
Use whatever tools are available to observe the live application:
- Screenshots — Capture rendered UI states (screens, components, transitions)
- HTTP responses — Status codes, response bodies, headers from live endpoints
- Console/log output — Application logs, warnings, errors during interaction
- Interaction results — Outcomes of taps, clicks, navigation, form submissions
- State changes — Database writes, file system modifications, side effects
Evidence must come from the running app. "The code looks correct" is not evidence.
"I hit GET /users and received a 200 with a JSON array of 3 users" is evidence.
Evidence Format
For each piece of evidence, record:
- Action taken — What you did (e.g., "Tapped the Submit button")
- Expected result — What the spec says should happen (e.g., "Form submits and shows success toast")
- Observed result — What actually happened (e.g., "Form submitted, success toast appeared after 200ms")
- Judgment — Pass or fail, with reasoning if fail
Structure evidence per dimension. Each dimension's evidence field should reference
specific runtime observations, not code impressions.
Stack-Specific Patterns
These are guidance for common stacks, not hard requirements. Use whatever tools
are available in your environment.
Flutter
Use the marionette MCP tool to interact with the running app:
- Navigate between screens
- Tap buttons, scroll lists, enter text in fields
- Take screenshots of each state
- Verify visual output matches spec
Go API
Use curl or the go-dev-mcp tool against the running server:
- Hit endpoints with valid and invalid inputs
- Verify status codes, response shapes, headers
- Check error responses for proper structure
- Test concurrent requests if relevant
Web Frontend
Use browser tools or curl against the running app:
- Load pages and verify rendering
- Submit forms, click links, test navigation
- Check responsive behavior at different viewport sizes
- Verify client-side state management
Failure Classification
Not all failures mean the implementation is bad. Distinguish between two categories:
App Failure — Score It
The application itself is broken. This is evaluation-relevant:
- App crashes on launch → Functionality/Correctness score 1-2
- Endpoint returns wrong data → Correctness impacted
- UI doesn't render spec'd components → Visual Quality / Domain Spec Fidelity impacted
Score these honestly using the domain template's scale definitions.
Harness Failure — Escalate, Don't Score
The test infrastructure broke, not the app (runtime didn't start, port unreachable,
MCP tool unavailable, environment misconfigured).
Do not penalize the implementation for harness failures. Report the failure to
the orchestrator and request re-evaluation with a working runtime.
