softprobe-ai-test-sre

name	softprobe-ai-test-sre
description	Diagnose failed integration and E2E test sessions with Softprobe. Use when an AI coding agent needs to authenticate Softprobe, add Sessify session correlation, query DuckLake evidence locally with DuckDB, inspect session-scoped traces/logs/metrics, or optionally replay selected dependency behavior.
when_to_use	Trigger for requests mentioning Softprobe, softprobe, failed integration tests, failed E2E tests, Sessify, x-softprobe-session-id, session_id, DuckDB, DuckLake, traces, logs, metrics, or optional replay.

Softprobe AI Test SRE

Softprobe is designed for AI coding agents. Human engineers normally bootstrap access, then ask their agent to use this skill.

Required Context

Before editing code or diagnosing a failure, load the canonical agent context:

1. Online when available: https://docs.softprobe.dev/ai-context.md
2. In the Softprobe monorepo: docs-site/public/ai-context.md

For schema details, read references/SCHEMA.md before writing SQL.

Bootstrap Checks

Use machine-readable commands first:

softprobe doctor --json

If credentials are missing, tell the engineer to run:

softprobe login
softprobe skill install
softprobe data setup --json

softprobe data setup writes connection material under ~/.softprobe/data/<profile>/. Do not ask the engineer to paste secrets into the repo.

Session Correlation

The session_id is the primary investigation key.

When instrumenting tests, prefer the existing client session SDK such as Sessify. The test should create or receive a Softprobe session id and send it on inbound app traffic as:

x-softprobe-session-id: <session id>

Normal Playwright, Cypress, Jest, pytest, and integration test runners keep owning execution. Softprobe adds session correlation and evidence capture.

Local DuckDB Diagnosis Workflow

Use local DuckDB as the query engine. The runtime brokers connection material; it is not the agent SQL execution engine.

1. Read ~/.softprobe/data/<profile>/.env. Use default when the profile is not specified.
2. Read references/SCHEMA.md.
3. Start with session-scoped SQL by session_id.
4. Run the query helper from this skill directory:

python3 scripts/query_ducklake.py --env ~/.softprobe/data/default/.env "SELECT * FROM traces WHERE session_id = 'SESSION_ID' ORDER BY timestamp;"

If the helper reports missing Python packages, install them in the agent's local Python environment:

python3 -m pip install duckdb pytz

Canonical first query:

SELECT *
FROM traces
WHERE session_id = 'SESSION_ID'
ORDER BY timestamp;

Then query logs and metrics only as needed:

SELECT timestamp, severity_text, body, trace_id, span_id
FROM logs
WHERE session_id = 'SESSION_ID'
ORDER BY timestamp;

Use evidence to classify the failure as a real regression, flaky test, bad test data, environment issue, or incorrect test expectation.

Instrumentation Guidance

When asked to instrument a repository:

• Detect the language, framework, and test runner before editing.
• Add OpenTelemetry instrumentation where the service needs traces/logs/metrics.
• Add Sessify or the approved client session SDK to tests so each failing test has a session id.
• Ensure test traffic reaches the app through the intended ingress or proxy path with x-softprobe-session-id.
• Run softprobe doctor --json before deeper runtime or header debugging.

Do not make tests hand-write raw /v1/sessions, /v1/inject, or /v1/traces HTTP calls unless the user explicitly requests protocol-level tests.

Optional Replay

Replay is optional. Use it only when the diagnosis needs controlled dependency behavior or the user explicitly asks for replay testing.

For replay work, use SDK helpers such as findInCase and mockOutbound, load committed case files, and keep mocks focused on outbound dependencies. Do not make replay the first diagnostic path.

Guardrails

• Do not ask the runtime to execute arbitrary SQL for the agent.
• Use local SQL through DuckDB with the .env written by softprobe data setup.
• Do not commit .env, .env.*, .venv, __pycache__, local DuckDB files, credentials, or generated package artifacts.
• Keep diagnosis evidence-backed and cite the session id plus the queried rows or fields that support the conclusion.