exploring-scouts

# Exploring Signals scouts A **scout** is a scheduled agent that wakes on its own interval, looks at one PostHog project, decides what's genuinely worth surfacing, and either writes it into the Signals inbox as a **report** or closes out empty (a real, valid outcome). PostHog ships a fleet of canonical scouts — a cross-product generalist (`signals-scout-general`) plus per-surface specialists (error tracking, logs, AI observability, experiments, feature flags, session replay, web analytics, surveys, and more). A project may also have **custom scouts** beyond the canonical fleet — any `signals-scout-*` skill a team authored (e.g. `-brand-mentions`, `-mcp-feedback`) shows up here too, so don't assume a fixed roster: `scout-config-list` is the authoritative roster for a project. (One caveat: a just-authored scout has no config row until the coordinator's next tick auto-registers one — or until someone registers it via the write-side `scout-config-create` — so a brand-new scout may briefly be missing from the list.) This skill helps you **understand and explore what a project's scouts are doing and how they're performing** — entirely through read-only MCP tools. It is the observability counterpart to the `authoring-scouts` skill (which teaches writing and tuning) and to the `inbox-exploration` skill (which covers the inbox reports scouts feed into). (The scout tools were recently renamed from `signals-scout-*` to `scout-*`; if a `scout-*` name comes back unknown, the server may still expose it under the legacy `signals-scout-*` name — search the tool catalog and call whichever name it returns.) **A scout's output is inbox reports, written 1:1.** Scouts list `emit_report` / `edit_report` in their `allowed_tools` and **author or edit inbox reports directly**; a run's output shows up as **`emitted_report_ids`** (reports it authored) and **`edited_report_ids`** (reports it updated). The run rows also carry `emitted_count` / `emitted_finding_ids` — **legacy fields from the deprecated signal-emitting channel** (weak `emit_signal` findings a pipeline consolidated). On a report-channel scout they stay `0` / empty even on a productive run; a non-zero tally means the run came from a scout still on the legacy channel (an old custom scout, or a canonical scout not yet ported) — real output for that run, not noise. When unsure of a scout's channel, check its `allowed_tools` via `skill-get`. **Never read `emitted_count: 0` as "did nothing"** — check the report columns and the run summary first. A scout whose config carries a `structured_output_schema` has a third output channel next to reports: schema-validated **measurement records**, recorded as `$scout_structured_output` events in the project (only _scalar_ top-level payload keys flatten to `output_<key>` properties — object and array fields live solely inside the full `output` property, so a missing `output_<key>` is not a missing value; `subject` names the judged entity) rather than as run-row columns. The events are the ground truth — `metadata.derived.has_structured_output` says the run had at least one batch **accepted**, which is a fast per-run screen but not delivery confirmation (a rare capture failure after acceptance leaves it true with fewer or no events behind it), so count the events when the number of records matters. See [`references/scout-data-model.md`](references/scout-data-model.md) for the event shape. Of the scouts PostHog ships, `signals-scout-mcp-tool-calls` is the one that records a series: every run that finds MCP traffic writes a per-category roll-up of MCP tool health and per-tool session-share rows, whether or not it files a report, so a healthy category is distinguishable from an unmeasured one. No records is not by itself a broken channel. A run on a project with no recent `$mcp_tool_call` events stops early and records nothing, and a dry-run scout (`emit` off) has nowhere to record to. Check the run summary, the `not-in-use:mcp_analytics` scratchpad entry, and the config's `emit` before you call the channel broken. Find them with `event = '$scout_structured_output'` and `properties.skill_name = 'signals-scout-mcp-tool-calls'`; the scout's own `references/metrics-dashboard.md` carries the chart and alert recipes. Each run also carries a `metadata` map. Top-level: the provenance set `harness_prompt_version` / `report_channel` (`none`, `emit`, `edit`, or `both`) / `skill_origin` / `github_guidance`, saying which instructions the run was given; plus routing keys (`model` / `runtime_adapter` / `reasoning_effort` / `service_tier`) only when a gate or pin overrode the default. Nested under `metadata.derived`: booleans the harness computes at the end of the run (`has_emit_report`, `has_edit_report`, `has_self_improvement`, `has_chart`, `has_self_validation`, `has_structured_output`, `has_not_in_use_closeout`). When comparing runs (before/after a prompt change, one model against another), segment on all four provenance values first: runs differing on any of `harness_prompt_version`, `report_channel`, `skill_origin`, or `github_guidance` were given different instructions and aren't a like-for-like population. Runs predating this field have none of them, so treat missing provenance as unknown and exclude those runs from a comparison rather than pooling them. For "what kind of run was this?" questions — did it author a self-improvement report, did it validate its follow-up queue — read `derived` rather than parsing the prose summary. It's computed server-side from what the run actually did, so it can't disagree with the run's own output — with one exception: `has_structured_output` tracks batches the run had **accepted**, not events delivered, so it alone can be true with fewer or no records behind it (count the events, as above). No `derived` map at all means unknown, not "all false" — the run predates the field, failed before finishing, or its stamp failed. Most runs from before this shipped have no map, so don't read their absence as a finding. To find **pointless runs**, read `has_not_in_use_closeout`. It is true when the run wrote a `not-in-use:` scratchpad entry (its quick close-out found the product off) and produced no report, edit, finding, or structured-output record. The harness counts the write against the run whose sandbox token made it, so a sibling scout's close-out on the same team is not credited to this run. A quiet run that looked and found nothing reads `false`. The pointless-run rate of a scout is the share of its finalized runs (runs that carry a `derived` map) with this flag true. Use it to rank which scout to gate with a pre-check next, and to confirm that a shipped pre-check moves the rate toward zero. Runs from before this flag shipped have no `has_not_in_use_closeout` key, so exclude them rather than count them as false. There are six things you can observe about the fleet, each with its own tool: | What you want to know | Tool | What it tells you | | -------------------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Which scouts run, how often, in what posture | `scout-config-list` | One row per scout: schedule (`run_interval_minutes` or `run_cron_schedule`), `enabled`, `status` / `pause_reason`, `emit`, `output_destinations`, `write_scopes`, `mcp_gateway_server_ids`, `last_run_at`, `consecutive_failure_count`, `description`, `scout_origin`, `owners`, `tags` | | What the scouts actually did, run by run | `scout-runs-list` / `-retrieve` | Per-run status, timing, end-of-run summary, `emitted_report_ids` / `edited_report_ids`, `error` / `failure_reason` on a failed run, deep-link; scope with `skill_name` | | What the fleet has learned across runs | `scout-scratchpad-search` | Durable per-team memory (baselines, noise, allowlists) | | What the team has told the fleet | `scout-notes-list` | Steering notes humans/agents left for scouts (per-scout or fleet-wide, newest first) | | Which reports a run wrote or edited | the run row itself | `emitted_report_ids` / `edited_report_ids`; resolve each id via `inbox-reports-retrieve` | | What the scouts surfaced to the user | `inbox-reports-list` | The scout-written reports, as the user sees them (`scout: "<skill_name>"` for one scout; `source_product: "signals_scout"` for the fleet) | `scout-config-list` takes a `tags` parameter (comma-separated) to narrow the roster to the scouts carrying any of the given labels — useful on a large fleet when the question is scoped to one area, e.g. `tags=revenue`. `scout-runs-list` takes `skill_name` (and optionally `skill_version`) to scope the dump to one scout, which is the normal way to answer any question about a single scout without paging through the fleet. The orienting tool is `scout-project-profile-get` — the deterministic snapshot of "what's true about this project" that every scout cold-starts from. When a scout found nothing, this is usually why. ## Output handling: expect to offload to a file Two of these tools — `scout-runs-list` and especially `tasks-runs-session-logs-retrieve` — routinely return payloads that **overflow an MCP client's token budget and get spilled to a file**. This is the normal path, not an error. Plan for it up front rather than discovering it after a failed call: - **Keep `limit` small** on `scout-runs-list` (~10–15). Each row carries a long prose `summary`, and runs come back newest-first across the _whole_ fleet, so even a modest page is large. - **Session logs are large by nature.** A single run's log is hundreds of KB to a few MB. Fetch it with **`call --json`** (so the saved file is real JSON, not the pretty text format — `jq`-able) and read the saved file with `jq` / a script rather than inline. - **Don't hand-parse the session log.** The bundled [`scripts/`](#helper-scripts) do the reconstruction for you — see below. ## Start here: is the fleet even set up? Don't assume the project has scouts. The fleet only runs on teams enrolled via the `signals-scout` feature flag, and a project may have no configs, all-disabled scouts, or scouts stuck in dry-run. Run this first whenever a user asks about their scouts for the first time in a session. ```json scout-config-list ``` Read the result against three cases: The config list is unpaginated — it comes back as `{ results: [...] }` (a bare array), with no `count` field. Read the result against three cases: - **Empty (`results: []`)** — no scouts are registered. The project isn't enrolled in the scout fleet (or hasn't ticked yet). Say so plainly; don't go fishing for runs. Point the user at the Signals scout settings / PostHog Desktop onboarding rather than inventing activity. - **Configs exist but all `enabled: false`** — the fleet is registered but paused. Nothing is running. Tell the user which scouts exist and that they're all off — and say who switched each one off, which `status` carries: `paused_by_user` means a person (or a launch seed posture) turned it off, `paused_by_system` means an automatic pause with its cause in `pause_reason` (`no_output` / `ignored` / `repeated_failures`). Either kind resumes with `enabled: true` via `scout-config-update`. A `repeated_failures` pause is the failure breaker: the streak of scheduled failures (`consecutive_failure_count`; manual and workflow-triggered runs don't count) ran one past what the schedule fits in twelve hours, clamped to 5–25 (daily: 5, rolling hourly interval: 13, hourly cron: 15, since cron slots are counted over a window padded for daylight-saving shifts). It is half-open, so the coordinator probes the scout once a day and resumes it on a clean run (unless the project is at its enabled-scout cap, which leaves the row paused after a clean probe); read the newest run's `failure_reason` to say what kept failing. - **At least one `enabled: true`** — the fleet is registered and that scout is allowed to run. For each enabled scout note its cadence (`run_cron_schedule` when set, else `run_interval_minutes`; the cron wins), `emit` (false = **dry-run**, runs but writes nothing to the inbox), and `last_run_at`. A `status` of `pending_pause` means the scout still runs but the system has flagged it to pause soon (cause in `pause_reason`); any config edit clears the warning. One caveat before reporting "it's live": runs are gated by the `signals-scout` feature flag, not by `enabled`. A project that was enrolled and later drained from the flag keeps its `enabled: true` rows, but the coordinator no longer plans runs for it — so a stale or `null` `last_run_at` on an enabled scout usually means the project is no longer enrolled, not that the scout is idle. **`last_run_at` is a _dispatch_ stamp, not proof a run executed.** The coordinator advances it the moment it _enqueues_ a child workflow for a due scout — before any worker picks the run up. Child dispatch is fire-and-forget, so if workers are saturated or down the children just queue and no run ever materializes, yet `last_run_at` keeps marching forward each tick. So a recent `last_run_at` means "dispatched this tick," **not** "a run is genuinely happening." The authoritative liveness signal is the newest actual **run row** in `scout-runs-list`, not the config stamp. Cross-check them: if `last_run_at` is fresh but no run row has appeared for that scout beyond its expected schedule, check the recorded pre-check outcome before diagnosing queued or stranded runs. Don't report "it's running" off `last_run_at` alone. **A scheduled pre-check can skip without a run row.** First read `last_precheck_at` and `last_precheck_outcome` on the config, following `authoring-scouts` → `references/scheduled-precheck.md` → "Diagnose a tick with no run row". A recent recorded `skip` for the tick can explain quiet; stale or absent outcome data leaves the cause unknown. Keep `scout-config-precheck-test` for query testing: a check made now cannot establish what happened on an earlier tick, and your access can differ from the scheduled check's acting user. A skip does not establish overall scout health. A scout that is `enabled: true` but `emit: false` is the most common source of "my scout isn't doing anything" confusion: it _is_ running and reasoning every tick, it just isn't allowed to post reports yet. Always surface the `emit` posture when reporting on a scout. See [`references/scout-data-model.md`](references/scout-data-model.md) for every field on a config, run, and scratchpad entry, the run status values, and how the pieces link together. ## Workflow: survey the fleet "What scouts do I have / what are they doing?" — lead with `scout-config-list`, then enrich with the most recent run per scout so the user sees liveness, not just configuration. 1. `scout-config-list` — the roster. 2. `scout-runs-list` once with a small `limit` and pick the newest run per `skill_name` (runs come back newest-first across the whole fleet, so one call usually covers everyone); for a scout that doesn't appear in that page, call again with `skill_name` set. Report `status` and how long ago it ran. Present it as a table the user can scan — scout, cadence, posture, last run, last outcome — and call out anything anomalous (never run, last run errored, stuck in dry-run for a long time). ## Workflow: understand one scout end to end "How does my error-tracking scout work / how is it doing?" 1. **Read its config** — find the row in `scout-config-list` for `signals-scout-error-tracking`: schedule, posture, last run. 2. **Read its body** — `posthog:skill-get {"skill_name": "signals-scout-error-tracking"}` returns the team's actual instruction set (which may be a canonical default or a diverged, hand-edited row). This is what the agent is told to do every run — its signal-vs-noise discriminator, explore patterns, and disqualifiers. To understand _why_ a scout behaves the way it does, read its body. 3. **Read its recent runs**: `scout-runs-list` with `skill_name` set to the scout (add `text` to search its summaries for a topic). The end-of-run `summary` on each run is the scout's own account of what it looked at and decided; a failed run carries `failure_reason` instead. 4. **Read what it remembered** — `scout-scratchpad-search` (see below). The memory entries a scout wrote reveal the baselines and noise it has internalized about this project. 5. **Read what it was told** — `scout-notes-list {"skill_name": "signals-scout-error-tracking"}` returns the steering notes humans left for this scout plus the general fleet-wide ones — exactly what its runs read as prior context. Each note carries an `origin`: `human` for one left directly, or a derived kind the inbox forwarded automatically: `report_dismissal` (a dismiss/snooze note), `report_discussion` (a question typed into a report's Discuss box), `report_feedback` (a note left with a thumbs rating), `report_reviewer_correction` (someone added or removed a suggested reviewer). Derived notes expire after ~30 days, and the list hides them from a caller without report read access (`task:read`), since they quote report content, so a credential with only `signal_scout:read` sees the human notes alone. Notes addressed to `pipeline:report-research` steer the report pipeline's research stage, not this scout, so they won't appear here. A behavior change that doesn't trace to a skill edit or a scratchpad entry often traces to a note. When asked to _steer_ a scout with a note (rather than observe), hand off to the `authoring-scouts` skill, which covers the notes channel's write side. ## Workflow: read recent runs