token-harness

name

token-harness

description

Observe CLI model calls, prompt/input payloads, stdout/stderr, provider-reported usage metadata, and local token estimates across Codex, Claude, DeepSeek, OpenAI, Anthropic, or other AI CLIs. Use when a user needs truthful token visibility, prompt-origin accounting, usage audits, or a harness around model CLI input/output; labels exact provider usage separately from local estimates and blocks claims that cannot be proven.

Token Harness

Purpose

Observe model work from the MindShare Central prompt path first. Preserve the input, output, command metadata, and usage evidence needed to explain where tokens came from.

The primary integration point is MindShare Central's provider dispatch layer: sendCodexMessage, sendClaudeMessage, sendDeepSeekMessage, and Combo child calls. A separate PowerShell or external wrapper is only for model calls that happen outside Central.

This skill does not pretend local counting is billing truth. Exact token usage is only exact when the provider/runtime emits usage metadata. Local tokenizer counts are estimates and must be labeled as estimates.

Versioning

Current version: 0.2.0.

• Patch: wording, parser, or report-shape clarification.
• Minor: new provider adapter, trace sink, or output field.
• Major: changed run contract, log schema, or activation boundary.

Changelog

• 2026-07-03 - v0.2.0 - Added MindShare Central embedded dispatch integration; every Central office prompt now writes a local harness run without a separate shell step.
• 2026-07-03 - v0.1.0 - Initial harness skill with local run wrapper, transcript ingest, usage summary, and provider-usage truth boundary.

Authority Boundary

• Owner: Vik / Program Architect.
• Default authority: local observability only.
• Writes: .mindshare/token-harness/runs/** unless the user passes another --run-root.
• Reads: MindShare Central assembled prompts, provider adapter output, provider usage metadata when exposed, explicit prompt files, piped stdin, wrapped CLI stdout/stderr, and transcript/log files the user points at.
• Does not read secrets, shell history, provider accounts, billing portals, or hidden runtime logs unless the user explicitly authorizes that source.
• Does not send traces to LangSmith, Phoenix, OpenTelemetry, or any external sink without a configured sink and explicit approval.
• MindShare Central local dispatch activation is approved for the local client path. External wrappers, autonomous/scheduled capture, and external trace export remain gated.

Truth Model

Classify every number before reporting it:

• provider_reported: usage emitted by the model provider, SDK, runtime, or CLI. This is the highest-trust token source.
• local_tokenizer_estimate: tokenized locally from captured text. Useful for input attribution, not proof of billable usage.
• local_heuristic_estimate: fallback character-based estimate when no tokenizer is available.
• bytes_chars_only: exact payload size, but no token claim.
• unknown: hidden context, tool payload, system prompt, cached prefix, or provider-side expansion that was not exposed.

If the harness cannot prove a count, say so directly.

MindShare Central Embedded Harness

For prompts sent inside MindShare Central office chat, do not ask the user to run a separate command. Central starts and finalizes the harness from its own provider dispatch functions:

• Codex: captures the exact stdin prompt sent to codex exec --json.
• Claude: captures the exact stdin prompt sent to claude --print --output-format json.
• DeepSeek: captures the exact system/user message payload sent to the API.
• Combo: preserves the child Codex/Claude harness runs; it does not fabricate a combined provider count.

Each turn writes:

• metadata.json
• input.txt
• output.txt
• stdout.txt
• stderr.txt
• events.jsonl
• usage.json

The department chat token line surfaces the latest run evidence. Click it to reveal the local usage.json file. Provider-reported usage is exact only when the provider adapter returned a usage object; otherwise the run is labeled estimate_only.

Run A CLI Under The Harness

Use scripts/token_harness.py run only when the next model call happens outside MindShare Central and can be launched from a command.

python skills/token-harness/scripts/token_harness.py run \
  --provider codex \
  --model gpt-5.5 \
  --prompt-file prompt.txt \
  -- codex exec

The script creates a run directory containing:

• metadata.json: command, provider, model, cwd, start/end time, exit code.
• input.txt: captured prompt/stdin when available.
• stdout.txt and stderr.txt: wrapped process output.
• events.jsonl: start, stream, and end events.
• usage.json: byte/char counts, local token estimates, provider usage records found in output, and accuracy notes.

Use --run-root <path> to write somewhere other than .mindshare/token-harness/runs.

Ingest Existing Logs

Use ingest when the model call already happened and the user provides transcript/log files.

python skills/token-harness/scripts/token_harness.py ingest \
  --provider claude \
  --model unknown \
  --input-file prompt.txt \
  --output-file transcript.txt \
  --out usage.json

Read references/provider-usage-contract.md before adding a new provider adapter or claiming exact usage from a transcript format.

Summarize Runs

Use summarize to aggregate local harness runs.

python skills/token-harness/scripts/token_harness.py summarize \
  --run-root .mindshare/token-harness/runs

Report:

• How many runs have provider-reported usage.
• How many runs are estimate-only.
• Provider-reported totals when available.
• Local estimated input/output tokens for attribution.
• Unknown or uncaptured input surfaces.

LangSmith / Phoenix Fit

Use LangSmith, Phoenix, or OpenTelemetry as trace stores only after the local harness produces a clean event record. Attach provider usage payloads as trace attributes. Do not infer true usage from trace text unless the provider usage object is present.

Minimum trace fields:

• run_id
• provider
• model
• role
• input_sha256
• output_sha256
• provider_reported_usage
• local_token_estimates
• accuracy_class

External trace export is secrets-gated and activation-gated.

Output Standard

Every token visibility report must include:

1. Source of truth: provider usage, local tokenizer estimate, heuristic estimate, or unknown.
2. Input attribution: prompt file, piped stdin, wrapped interactive stdin, transcript ingest, or uncaptured.
3. Hidden-token caveat: system prompt, tool payloads, memory injection, RAG context, cache behavior, and provider-side formatting may exist outside captured local text.
4. Next correction: wrapper adoption, runtime adapter, transcript source, or provider billing/API export needed to close the gap.

Failure Conditions

Stop and report a blocker when:

• The user asks for exact billable usage but no provider usage metadata is available.
• The CLI requires a raw TTY and cannot be safely wrapped by the line proxy.
• The required transcript contains secrets or private data outside the approved scope.
• A proposed integration would send prompts or outputs to an external service without approval.