| name | analyze-dataset-token-length |
| description | Analyze the token length of an OT-Agent conversation-format (ShareGPT-style) dataset — the per-trace distribution (median/p90/max) and/or counts under a token threshold + a metadata predicate (e.g. "task_complete AND < 32768 tokens"). Use when asked how long traces are, how many fit a context window (32k/131k), or to filter a trace dataset by length + a field. Uses the OT-Agent analysis tools + the Qwen3-8B tokenizer. Runs LOCALLY on the Mac (no GPU); full-dataset tokenization of ~10k multi-turn traces takes a few minutes → run it in the background. |
analyze-dataset-token-length
OT-Agent trace datasets are conversation-format (ShareGPT-style): each row is
{"conversations": [{"role","content"}, …], + metadata} (some use "messages"; metadata
fields are e.g. task, result, run_id, trial_name, model, agent). "Token length
of a trace" = the tokenized length of the whole conversation.
The canonical OT-Agent tools (don't reinvent)
scripts/analysis/utils.py — canonical pure conversation/token helpers:
extract_conversation_text(record), render_token_representation(...), and
count_conversation_tokens(...). Every count must explicitly select
serialized, conversation_text, or chat_template; these are different
measurements and must never be silently substituted for one another.scripts/analysis/context_length_compare.py — cross-dataset context-length comparison.
Tokenizer convention
Always Qwen/Qwen3-8B (AutoTokenizer.from_pretrained("Qwen/Qwen3-8B", trust_remote_code=True)).
Our trace datasets are Qwen3-8B-tokenized even when named for GLM/Kimi/etc. — those "GLM-4.7-…"
models are Qwen3-8B SFTs (see memory reference_glm47_swesmith_is_qwen3_8b); the served model
name in a row's model field (e.g. hosted_vllm/<numeric-id>) is NOT a usable tokenizer name.
Three token-count representations — pick by the question
- conversation_text =
tokenizer(extract_conversation_text(row), add_special_tokens=False) — fast;
slightly under-counts vs training (no chat-template tokens).
Right for distribution/relative comparisons.
- chat_template =
len(tokenizer.apply_chat_template(conv, tokenize=True, add_generation_prompt=False))
— what an SFT trainer actually tokenizes; use when the question is "does it fit a 32k/131k
training window." If a trace's role shape makes the template raise, report it as
uncountable for this representation and tally it separately; do not substitute a
plain-text count.
- ⚠️ The two can differ by MORE than the wrapper tokens — and in the surprising direction.
Qwen3's chat template strips historical
<think> blocks from earlier assistant turns, so on
thinking-mode traces apply_chat_template can count fewer tokens than plain-concat (which keeps
all thinking) — i.e. more traces "fit" under the template. So the "right" count for a < N filter
depends on whether your SFT template preserves thinking: default Qwen3 (strips) → optimistic
count; a thinking-preserving template (qwen3_thinking_acc.jinja2) → conservative count ≈ plain.
Report BOTH and pick by the training template; for a safe "fits 32k" answer use the larger (plain /
thinking-preserving) count.
- serialized = compact JSON of the raw
conversations value, including role
fields and JSON punctuation. This is the legacy datagen-counter measurement;
it is useful for continuity but is neither plain text nor training-faithful.
Threshold + metadata filter-count (the common ask)
Recipe: load_dataset (non-streaming) → per row compute (a) the token count and (b) a metadata
predicate → count the intersection; report each leg separately so it's auditable.
⚠️ The metadata-confound trap (read this before any field predicate)
Instruction text leaks into the trace. Fields like task_complete appear verbatim in the user
instruction of EVERY trace (…include "task_complete": true in your response…), so a naive
'"task_complete": true' in full_text matches all rows (false 100%). Scope the predicate to the
agent's actual emission — i.e. an assistant-role message containing the field, not the prompt:
def agent_complete(conv):
return any(m.get("role") == "assistant" and '"task_complete": true' in (m.get("content") or "")
for m in conv)
Always sanity-check the predicate VARIES (not all-true / all-false) before trusting a count — print
the per-leg breakdown and an early per-1000-row progress line. (Same caution for any tool-name /
status substring: confirm you're matching the agent's output, not the system/user scaffolding.)
How to run
Local, otagent python, HF token sourced; full-dataset tokenization of ~10k multi-turn traces is a
few minutes → background it:
source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"
/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python scripts/analysis/<script>.py
First run downloads + caches the parquet (~hundreds of MB). A benign 'NoneType' has no attribute 'ArrowInvalid' on streaming-generator teardown can be ignored (use non-streaming load_dataset anyway).
Worked example — swesmith, "task_complete AND < 32768 tok"
DCAgent2/GLM-4.7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k (9437 traces):
detect completion via the assistant-scoped "task_complete": true (not the instruction prose),
count tokens via apply_chat_template (Qwen3-8B), filter complete AND ct < 32768. Reports three
legs — #complete, #<32k, and the intersection — so the filter is auditable. (Early progress
1000/9437: complete=924 / fit32k=886 / both=854 confirmed the predicate varies ≈92%, i.e. the
confound was correctly excluded.)
