nemo-compass

# NeMo Compass ## Before running Compass reads telemetry; it cannot create it. Confirm all three: - The target agent already has traces in Intake. No traces means no Insights. - The platform is reachable at `NHX_BASE_URL`. - The CLI has default and fast Platform models configured through `nemo setup`, or the submission supplies explicit model references. The Platform must have Jobs and an `agents.execute` runtime available to execute the analysis. An `ETHOS.md` file is optional. It gives Compass the agent's intent, constraints, and success criteria. Code and traces don't contain that context. Without it, Compass can only judge an agent against itself. ## What it produces Analyze an agent's behavior from its own telemetry and record what recurs as Insights. An Insight is a persistent, named description of one recurring problem, and it is the unit of work the rest of the optimization loop runs on. Each carries: - `title` — a sentence naming the failure, such as "Retrieval drops relevant context near the token limit" - `description` — the failure mode, the tool or model call it affects, and the conditions that trigger it - `evidence` — supporting trace IDs with optional URLs and span references, so a developer can audit the reasoning and build regression tests Compass targets at least three representative traces per Insight and appends evidence to an existing Insight rather than filing a near-duplicate. It judges behavior rather than status or scores, so it finds failures in sessions that reported success and passed their evaluations. Two well-evidenced Insights are worth more than ten vague ones, so a run that files nothing is a valid outcome. ## Run it Replace the quoted placeholders below with the target agent and workspace. ```bash nemo compass analysis-runs create --agent "<agent-name>" --workspace "<workspace>" --wait ``` Add `--ethos ETHOS.md` to supply the agent's intended behavior, `--since` for an ISO-8601 lower time bound, or `--evaluation-id` to select an evaluation. The command submits an AnalysisRun backed by an `agents.execute` job. Expect several minutes. `--wait` exits non-zero unless the job completes successfully. The CLI supplies the configured default and fast models. Run `nemo setup` if they are missing, or pass `--default-model` and `--fast-model` explicitly. ## Where Insights are stored Compass creates and updates insights in NeMo Helix after analysis. Stored insights appear in Studio's optimizer view for the workspace. ## Verify Replace the quoted placeholders with the submitted run name and workspace: ```bash nemo compass analysis-runs get "<run-name>" --workspace "<workspace>" ``` Confirm the job completed and inspect its analysis report. A completed run may produce no new insights. When it records insights, read them back and check for a clear title, an actionable description, and non-empty `evidence`. Listing all insights for the agent can include earlier runs. ```bash curl --fail-with-body \ "$NHX_BASE_URL/apis/compass/v2/workspaces/<workspace>/insights/<insight-id>" ``` On an authenticated platform pass the token through curl's config, not argv. Replace `<url>` with the insight URL above: ```bash printf 'header = "Authorization: Bearer %s"' "$(nemo auth token)" | curl --fail-with-body -K - "<url>" ``` ## When it finds nothing Besides a real "nothing worth filing", three things produce an empty result. Scoping: Compass reads only what `--agent` and `--workspace` together select, and `agent_name` is carried on agent-level spans, not on their model and tool children. Volume: too few traces looks the same as a healthy agent. And telemetry that captures only the shape of a run, spans without the inputs and outputs, leaves nothing to judge however many spans there are.