examples-qa

name	examples-qa
description	Verify Instructor behavior through the repository's ./examples/ suite in pass, live record, or hermetic replay mode. Use when running selected examples or the corpus, capturing and reusing recorded LLM HTTP responses, diagnosing hub results, or distinguishing real errors, assertion failures, skipped examples, and live-only flaky outcomes.

Instructor examples QA

Use the examples suite as an end-to-end check of Instructor behavior, but keep the execution mode explicit. A live run tests provider and transport integration; a record run creates reusable HTTP fixtures; a replay run tests the current code against fixed provider responses without spending tokens or contacting a network.

Inspect before running

Run commands from the repository root. Discover the target and its canonical selector first:

php bin/instructor-hub list
php bin/instructor-hub show getters_and_setters

The hub accepts a front-matter docname, numeric index, short ID such as xe4bb, or example path. Read the target's run.php before diagnosing it. Check its tags and whether it builds an HTTP client implicitly or supplies a custom client/mock.

Treat these sources as authoritative when the prose documentation differs:

• just/examples.just — preferred record/replay command wrappers
• examples/boot.php and examples/_support/HttpRecordingBoot.php — mode and recording-directory behavior
• packages/hub/src/Commands/EnhancedRunAllExamples.php — corpus filtering and OK/ASSERT/ERROR/FLAKY classification
• packages/hub/src/Services/ExampleRepository.php — selector resolution
• QUALITY.md — repository QA policy and operational caveats

Choose the execution mode

Use the smallest mode that answers the question:

Mode	Command shape	What it proves
pass	INSTRUCTOR_EXAMPLES_HTTP=pass ... or no mode	Normal behavior; no record/replay middleware is attached. LLM examples use the live provider.
record	just examples-record <selector>	Live provider call succeeds and the response can be persisted as a sanitized cassette.
replay	just examples-replay <selector>	Current code consumes the fixed response correctly, with no network or provider credentials required.

Run without record/replay

Use pass mode for a normal baseline or a live provider smoke check:

INSTRUCTOR_EXAMPLES_HTTP=pass \
  php bin/instructor-hub run getters_and_setters

Omitting INSTRUCTOR_EXAMPLES_HTTP has the same effect because pass is the default. Pass mode attaches no ambient middleware. LLM examples therefore need their normal provider keys and network access; mock-only examples remain local.

For a code change, prefer this sequence:

1. Run a narrow live or existing-fixture baseline when provider integration is in scope.
2. Record only the affected examples if the fixture is missing or intentionally refreshed.
3. Replay the same selector to verify deterministic behavior.
4. Run the replay gate for the affected group, then the full replay corpus when practical.
5. Run package tests separately when the example result does not identify the failing layer.

Do not treat a replay pass as proof that provider authentication, DNS, TLS, rate limits, or current provider behavior work. Do not treat a live LLM failure as a product regression until it has been reproduced with replay or an appropriate unit/integration test.

Record one example

The preferred scratch-fixture workflow is:

just examples-record getters_and_setters

The wrapper sets:

INSTRUCTOR_EXAMPLES_HTTP=record
INSTRUCTOR_EXAMPLES_RECORDINGS_DIR=examples-recordings

The live response is stored under a namespaced directory such as:

examples-recordings/A01_Basics/BasicGetSet/
├── cassette.json
└── interactions/000001/
    ├── interaction.json
    └── response.body                 # or response.chunks.ndjson

Record the whole corpus only deliberately; it requires keys for every provider used and can be expensive:

just examples-record-all --limit=20
just examples-record-all --filter=errors

To create fixtures in each example's co-located recordings/ directory, omit the recordings-directory override:

INSTRUCTOR_EXAMPLES_HTTP=record \
  php bin/instructor-hub run getters_and_setters

The direct hub sugar is equivalent and is useful when keeping the mode visible at the invocation site:

php bin/instructor-hub run getters_and_setters --record
php bin/instructor-hub all --record --limit=20

The bin/instructor-hub launcher consumes --record, --replay, and --recordings-dir=<dir> before dispatching the hub command. If using a custom directory, use the same directory for record and replay:

php bin/instructor-hub run getters_and_setters \
  --record --recordings-dir=examples-recordings

Recording is live. Confirm the relevant provider key exists, expect token/network cost, and do not record broad personal or production data. Common credentials are sanitized before persistence, but prompts, model outputs, PII, and provider-specific secrets still require review.

Streaming recordings are published only after the stream completes naturally. An interrupted or failed stream must not be treated as a valid fixture.

Replay selected examples

Replay from the same root used for recording:

just examples-replay getters_and_setters

Or use the direct form:

php bin/instructor-hub run getters_and_setters --replay \
  --recordings-dir=examples-recordings

For keyless replay through a raw example path:

INSTRUCTOR_EXAMPLES_HTTP=replay \
INSTRUCTOR_EXAMPLES_RECORDINGS_DIR=examples-recordings \
php examples/A01_Basics/BasicGetSet/run.php

Replay is hermetic by default for requests that enter the configured ambient middleware. It is not a blanket network sandbox: examples that construct their own client can bypass the hook and need the no-replay treatment or separate inspection. For intercepted requests, the examples boot hook provisions dummy provider keys only so configuration resolution succeeds; the recorded response should be served before any provider driver can authenticate or open a network connection. A missing recording, request mismatch, exhausted ordered session, corrupt payload, or unsupported cassette version is a diagnostic failure, not a reason to fall back to the live provider.

Replay the corpus with optional controls:

just examples-replay-all --limit=20
just examples-replay-all --filter=errors
php bin/instructor-hub all --replay --dry-run
php bin/instructor-hub all --replay --filter=errors

--dry-run verifies the selected set without executing it. Use --filter=errors after a tracked run to focus on previously failing examples; use --force when the hub's status cache would otherwise omit an example.

Understand what is actually covered

The ambient hook in examples/boot.php attaches middleware through HttpClientDefaults only when the example's runtime resolves an implicit HTTP client. Examples that construct their own client, use a mock driver, or are marked no-replay may not exercise the cassette path. Inspect the example before calling an unchanged result evidence for record/replay.

Corpus filtering is intentional:

• skip: true examples are never selected.
• The broken tag skips known-broken examples in both modes.
• The no-replay tag skips examples that cannot be recorded/replayed when replay mode is active; the hub logs the skipped set.
• The flaky tag describes live LLM nondeterminism. It does not make a replay failure acceptable.

No individual example needs to be edited to enable the shared mode switch. Avoid bulk-editing the example corpus just to add record/replay setup.

Read hub output

For php bin/instructor-hub run, the hub executes one example and reports OK or ERROR; the detailed cause comes from the example subprocess and its ExecutionResult. For hub all, EnhancedRunAllExamples prints one row per example and a summary. Its result categories mean:

Output	Meaning	Gate effect
OK	The example process completed successfully, including its assertions.	Pass.
ASSERT	The example's assertion/expected-behavior check failed.	Real failure; inspect the assertion and current response.
ERROR	The process failed for a non-assertion reason: exception, missing cassette, provider/transport failure, syntax/runtime error, or infrastructure problem.	Real failure.
FLAKY	A live-mode example tagged flaky failed and the hub tolerated it.	Does not fail the live corpus gate, but is not proof of correctness.
skipped	The example was filtered by skip, broken, or replay-only no-replay.	Not evidence of a pass.

The flaky exception applies only to live mode. In replay mode the response is fixed, so a failure from a flaky-tagged example is counted as an error. This is the mechanism that turns a recorded flaky scenario into a deterministic regression check.

The all-examples exit code fails when real errors remain; tolerated live FLAKY rows do not fail it. The final summary and the per-example diagnostic line are more authoritative than a process that merely printed some expected-looking output.

Diagnose failures by source

Classify the failure before changing code:

Record-mode failures

• Provider authentication, DNS, TLS, rate limit, timeout, or HTTP errors are live integration failures. Check the key, provider availability, and request first.
• A stream that fails before natural completion should not produce a completed recording. Fix the live condition or record a controlled deterministic fixture; do not accept a partial cassette.
• A custom-client or mock example not creating a cassette is expected when the ambient hook is bypassed.

Replay-mode failures

• RecordingNotFoundException: the selected example has no fixture under the supplied root, or the request changed enough not to match. Confirm the exact root, namespace, selector, and whether the recording was made in stream mode.
• CassetteMismatchException: the next ordered interaction does not match the current method, URL semantics, body, stream mode, or response-shaping headers. Re-record only after deciding that the request change is intentional.
• CassetteExhaustedException: the example made more calls than the recorded session. Investigate changed retry/loop behavior; do not enable live fallback.
• CassetteCorruptException or unsupported-version errors: the fixture is damaged or incompatible. Restore/re-record/migrate it; do not reinterpret the bytes or silently call the network.
• A provider-key error in replay usually means the example bypassed examples/boot.php or the invocation did not propagate replay mode. Verify the entrypoint and environment before changing provider configuration.

Assertion and application failures

An ASSERT result means the example ran far enough to make its own correctness check and that check failed. Compare the current implementation with the fixed replayed response, then inspect the example's assertion and the relevant package tests. An ERROR with a stack trace or subprocess error is a runtime/infrastructure problem first, not automatically an Instructor semantic regression.

Inspect persisted diagnostics

Use the hub's tracked status after an all run:

php bin/instructor-hub status
php bin/instructor-hub status --detailed
php bin/instructor-hub status --errors-only
php bin/instructor-hub errors --list
php bin/instructor-hub show getters_and_setters

Use errors --run to re-run tracked failures only after choosing the intended mode, and errors --clear only when intentionally resetting the diagnostic history. The hub stores execution summaries and recent errors; those records explain what the hub observed, while the cassette files and example subprocess output explain whether the cause was request matching, provider transport, or Instructor logic.

For a suspected library regression, reduce the diagnosis to the narrowest evidence:

1. Reproduce the same example in replay mode.
2. Confirm the cassette is present and the failure category is stable.
3. Run the relevant package test or add a focused regression test.
4. Run the example again after the fix and confirm the error category disappears.

Do not convert an ERROR to a tolerated tag merely to make the gate green. Add or remove flaky, broken, and no-replay only when the example's documented behavior actually meets the tag's meaning and the change is reviewed.