| name | atai-newton-fusion-model |
| description | Call Newton's C 2.6 fusion model (`Newton::c2_6_8b_fp8_260424d7a55d5e`) on the prod `/query` endpoint with text, image, and video inputs in a single stateless request — no session, no batch job, no SSE plumbing. Use this skill when the user wants per-call multimodal reasoning (describe an image, summarize a clip, classify a state from sensor + visual context) without managing session lifecycle, or when they need the Newton C checkpoint that reasons over video frames via `/query`. Covers the request shape per modality, the two image-attachment paths (file_ids vs base64), multi-image mode, both video paths (`.mp4` + `max_frames`, and client-sampled frames + `query_metadata`), generation parameters, JSON-output prompting, latency budgets, and the C 2.6 identifier gotcha. Do NOT use for streaming / session-based activity monitoring, large multi-file batch jobs, or time-series embedding (KNN / anomaly) classification.
|
Newton C 2.6 Fusion Model — Multimodal /query in One Hop
Single stateless POST to /query for text, image, or video reasoning against the C 2.6 fusion checkpoint. No Lens session, no batch pipeline, no SSE. One call → one response.
When to Apply
- User wants to describe / classify / extract structure from an image with one API call
- User wants the same for a short video in a single
/query call
- User wants stateless multimodal reasoning (no per-session warmup, no orphan cleanup)
- User wants JSON output from a multimodal input (the prompt is the schema)
- User is building a serverless / per-request handler where a streaming Lens session would be the wrong shape
Do not use this skill when:
- The workload is live, streaming video → text from a long-running camera feed — a session-based activity monitor is the right shape, not stateless
/query
- The workload is a large multi-file batch job
- The task is time-series classification via embeddings (KNN / anomaly detection)
The Model Identifier
Newton::c2_6_8b_fp8_260424d7a55d5e
The full Newton:: prefix and the _fp8_ segment are required. Variants that omit either are rejected with 400 invalid_model_version:
c2_6_8b_fp8_260424d7a55d5e ❌ missing Newton:: prefix
Newton::c2_6_8b_260424d7a55d5e ❌ missing _fp8_
c2_6_8b_260424d7a55d5e ❌ both missing
This was probed end-to-end against https://api.u1.archetypeai.app/v0.5/query on prod.
Endpoint
POST {ATAI_API_ENDPOINT}/v0.5/query
Authorization: Bearer <API_KEY>
Content-Type: application/json
Both ATAI_API_KEY and ATAI_API_ENDPOINT are required — there is no default endpoint, so a wrong-deployment mistake fails loudly at startup instead of silently at query time. Prod is https://api.u1.archetypeai.app/v0.5. The same endpoint serves all three modalities below — what changes is the request body.
Wire Shapes
Text + JSON
{
"query": "Classify the operational state. Respond with ONLY the JSON object.",
"instruction_prompt": "You output {\"state\": \"...\", \"confidence\": <0..1>}.",
"file_ids": [],
"model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
"max_new_tokens": 300
}
- Put the system turn in
instruction_prompt. That is the only field C 2.6 honors — a directive sent in the legacy system_prompt field alone is silently ignored (verified against Newton::c2_6_8b_fp8_260424d7a55d5e: the same directive is obeyed in instruction_prompt and dropped in system_prompt), so omit system_prompt entirely.
- JSON output: put the schema in
instruction_prompt, the input in query, and explicitly tell the model not to wrap in markdown fences. The model otherwise often returns json ...
For reasoning over an attached text file: upload it as text/plain (a .txt file) via /v0.5/files, then put the filename in file_ids. The model then sees the file contents in its prompt context. Critical: this only works for text/plain. A file uploaded as text/csv (a .csv) is accepted and stored byte-identically, but its contents never reach the model on /query — it silently confabulates from priors (see the gotcha below and csv_vs_txt_proof.py). application/json uploads are rejected outright. So put tabular data in a .txt, or inline it in query.
See text_query.py for runnable demos of all three patterns. Its examples are framed as smart-home WiFi flow analysis; the attached flow log in example 3 is real data from the GHOST-IoT dataset (Anagnostopoulos et al., Sensors 2020, DOI:10.3390/s20226600) — see the script's docstring for full attribution.
Image
Two attachment paths, both reach the same fusion model — pick by use case — plus a flag for sending several images at once:
(a) Upload + reference by filename in file_ids — best when the same image is reused across queries (one upload, many /query calls):
curl -X POST $ATAI_API_ENDPOINT/v0.5/files \
-H "Authorization: Bearer $ATAI_API_KEY" \
-F "file=@dashboard.png;type=image/png"
{
"query": "Describe this image in three bullets.",
"instruction_prompt": "...",
"file_ids": ["dashboard.png"],
"model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
"max_new_tokens": 300
}
(b) Inline base64 in a data.base64_img event — best for one-shot calls where the upload roundtrip is wasted:
{
"query": "What is in this image? One sentence.",
"instruction_prompt": "...",
"file_ids": [],
"model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
"max_new_tokens": 200,
"events": [
{
"type": "data.base64_img",
"event_data": {
"contents": "<base64 string>",
"mime_type": "image/png"
}
}
]
}
(c) Multiple images — set multi_image: true for multi-image mode: each attachment is an independent image (before/after, multi-view, "is the same person in these images?"), not frames of a video. Platform limit: 16 images max per request. Without multi_image: true, an image list means "frames of one video" and requires query_metadata (see the Video section); lacking both, the request fails with 400 query_failed:
{
"query": "Image 1 is the bench BEFORE the task; image 2 is DURING. What changed?",
"instruction_prompt": "...",
"file_ids": ["assembly_before.png", "assembly_after.png"],
"model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
"max_new_tokens": 400,
"multi_image": true
}
See image_query.py for runnable demos of all three paths.
Video
C 2.6 reasons over video on /query two ways. This distinguishes it from the C 2.4 / 2.5 text checkpoints, which accept an .mp4 but ignore the frames and reply "I can't see videos."
(a) .mp4 + max_frames — upload the clip, and GPQ decodes and uniformly samples it server-side before the model sees the frames. No client-side video tooling needed:
curl -X POST $ATAI_API_ENDPOINT/v0.5/files \
-H "Authorization: Bearer $ATAI_API_KEY" \
-F "file=@clip.mp4;type=video/mp4"
{
"query": "Evaluate the assembly steps shown in the attached video.",
"instruction_prompt": "<task / inspection prompt>",
"file_ids": ["clip.mp4"],
"model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
"max_new_tokens": 500,
"max_frames": 32
}
(b) Client-sampled frames + query_metadata — send the frames yourself (one data.base64_img event each, or image file_ids), set multi_image: false, and include a query_metadata block with the video-metadata triple. The query_metadata block is what makes this work — without it the request fails with 400 query_failed:
{
"query": "This is a short video. Describe the sequence of actions in order.",
"instruction_prompt": "...",
"model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
"multi_image": false,
"max_new_tokens": 150,
"events": [
{"type": "data.base64_img", "event_data": {"contents": "<frame_0 base64>"}},
{"type": "data.base64_img", "event_data": {"contents": "<frame_1 base64>"}}
],
"query_metadata": {"raw_fps": 1.0, "frames_indices": [0, 1], "total_num_frames": 2}
}
Both shapes verified live (the file_ids variant of (b) works too). Notes:
max_frames — frames GPQ samples uniformly from the video before feeding the model. Default 32, platform max 64. Increase for longer / higher-detail clips; decrease to cap latency. Values above 64 fail silently: the request returns 200, but the video is dropped and the model confabulates a scene from priors (verified: max_frames: 80 on the assembly clip described "a person holding a smartphone"). Stay at ≤ 64.- Timeout — set the client request timeout to at least 600s. Video latency scales with
max_frames and clip length; default 30–60s timeouts will trip on non-trivial clips.
.mp4-direct on C 2.6 is a recent platform change. Earlier the .mp4 path silently dropped the frames for every model (a known GPQ bug), so older notes say "video doesn't work on /query." On C 2.6 today it does, via max_frames.multi_image selects between video and multi-image mode. multi_image: false (with query_metadata) = the attached images are frames of one video. multi_image: true = multi-image mode — the images are independent (before/after, multi-view), and query_metadata is not needed.
See video_query.py for runnable demos of both paths (mp4 direct, a max_frames tradeoff, and a frame-list demo) using a worker-assembly PASS/FAIL inspection prompt.
Generation Parameters
All C-model generations via /query accept sampling controls in the request body (verified accepted + honored on prod with C 2.6):
| Param | Default | Notes |
|---|
max_new_tokens | 256 | Hard ceiling on generated tokens (~4 chars each) — stops mid-sentence at the cap |
temperature | — | 0.0 = greedy/deterministic-ish; 0.7–1.0 for creative output. Capped at 2.0 |
do_sample | true | Master switch: false forces greedy decoding (temperature → 0, top_p/top_k stripped) — use for reproducibility |
repetition_penalty | 1.0 | Multiplicative logit penalty on already-emitted tokens; try 1.1–1.3 if the model loops on a phrase or repeats numbers |
presence_penalty | 0.0 | Additive penalty on any token that has appeared at least once (frequency-independent). Range ≈ −2..+2. C 2.6 only |
top_p | 0.8 | Nucleus sampling — keep the smallest token set with cumulative probability ≥ top_p |
top_k | — | Hard cap on candidate tokens per step (applied before top_p); 0 disables |
For "extract X"-style tasks, do_sample: false (or temperature: 0.0) plus a tight max_new_tokens is the right starting point.
Reproducibility caveat (verified): the sampling knobs are honored — on the same prompt, 5 runs gave 2 distinct outputs greedy (do_sample: false), 3 distinct at defaults, 5 distinct at temperature: 1.9 — but greedy is not bit-deterministic on this deployment (fp8 numerics / batching sit outside the decoder), contrary to the platform note that do_sample: false output "is deterministic given the prompt". On borderline judgments the residual wobble is enough to flip the answer: the same greedy assembly-inspection body returned all-PASS on one run and two FAILs on another. If a verdict matters, aggregate over a few calls rather than trusting one.
Latency Budgets
Observed against prod with the sample assets in references/sample_assets/:
| Modality | Body | p50 | Notes |
|---|
| Text only | {"query": "...", "file_ids": []} | ~500 ms | Confirmed during model-identifier probing |
| Text + JSON output | instruction prompt has schema | ~600 ms | Same shape, slightly longer with verbose schema |
Image via file_ids | one 1.9 MB PNG | ~1.3–1.9 s | observed (wind-turbines.png) |
| Image via inline base64 | same | ~0.8–2.5 s | base64 inflates the request; scales with image size |
Two images, multi_image: true | 2 × ~400 KB PNG | ~2.9 s | before/after comparison (assembly frames) |
Video, mp4 + max_frames=32 | ~31 s assembly clip | ~2–6 s | server-side decode/sample |
Video, 2 frames + query_metadata | 2 × ~400 KB PNG | ~1.9 s | client-sampled frame-list path |
Numbers are smoke-test order-of-magnitude. Production data may be 5–10× heavier and proportionally slower.
Response Shape
{
"query_id": "...",
"status": "completed",
"inference_time_sec": 0.42,
"query_response_time_sec": 0.49,
"response": {
"response": ["<the model output as a string>"]
}
}
Walk payload["response"]["response"][0] as the canonical extraction path. The reference scripts' extract_text() helper handles every shape variant we've seen.
Common Pitfalls
- Wrong model identifier.
c2_6_8b_fp8_260424d7a55d5e without Newton:: returns 400 invalid_model_version. So does the variant without _fp8_. Always use the full Newton::c2_6_8b_fp8_260424d7a55d5e.
- JSON wrapped in markdown fences. The model sometimes returns
json ... text.strip().removeprefix("```json").removesuffix("```")) or restate the no-fences rule in the user query as well as instruction_prompt.
- Video timeout. Default
requests.post timeout of 30–60s will fail on anything but trivial clips. Set timeout ≥ 600s.
- Multiple images without
multi_image: true need the query_metadata triple. A list of images defaults to "frames of one video" — but that path requires query_metadata: {"raw_fps": ..., "frames_indices": [...], "total_num_frames": N} in the body; without it the request fails with 400 query_failed (the error doesn't hint at the missing field). For independent images (before/after, multi-view), set multi_image: true instead — max 16 images, no query_metadata needed.
max_frames above 64 fails silently. The platform max is 64. max_frames: 80 still returns 200, but the video never reaches the model — it confabulates a scene from priors (same failure mode as the CSV gotcha below). Keep max_frames ≤ 64.- "Video doesn't work on
/query" is stale. Older notes say C 2.6 (and 2.4/2.5) can't see video. That was a GPQ bug that silently dropped .mp4 frames before any model. C 2.6 reads video now via .mp4 + max_frames — verified. The 2.4/2.5 text checkpoints still ignore frames.
file_ids takes the filename, not the fil_... uid. On upload the API returns both {"file_id": "<filename>", "file_uid": "fil_..."}. /query resolves file_ids by the filename (the file_id value, e.g. "dashboard.png"), not the fil_... file_uid. Passing the file_uid fails with a misleading 400 unsupported_file_type ("Found unsupported file type with file: fil_…") — it looks like a content-type problem but is really the wrong identifier. Verified: file_ids=["wind-turbines.png"] → 200; file_ids=["fil_…"] → 400.- Text attachments must be
text/plain, not text/csv. A .csv upload (text/csv) is accepted, stored byte-for-byte, and returns status: completed — but its contents never reach the model on /query; the model silently confabulates from priors (e.g. answers "TCP"/"port 80" for an all-SSH file). The identical bytes uploaded as .txt (text/plain) are read faithfully. There is no warning — the failure is silent, so you can only catch it by validating the answer. Upload tabular/text data as .txt, or inline it in query. Verify yourself with csv_vs_txt_proof.py (.txt reads 5/5, .csv 0/5 on identical content). application/json uploads are rejected with 400 invalid_file_type.
- Mixed-modality calls. You can attach both an image and a video file_id, but the fusion model's attention is finite. Stick to one modality per call for predictable results.
Local Setup
python3 -m venv .venv && source .venv/bin/activate
pip install -r skills/atai-newton-fusion-model/references/requirements.txt
cat > .env <<EOF
ATAI_API_KEY=sk_...
ATAI_API_ENDPOINT=https://api.u1.archetypeai.app/v0.5
EOF
python skills/atai-newton-fusion-model/references/text_query.py
python skills/atai-newton-fusion-model/references/image_query.py
python skills/atai-newton-fusion-model/references/video_query.py
The scripts are built on the official Archetype AI python client (pip install archetypeai): uploads go through client.files.local.upload(...), base64 encoding through archetypeai.utils.base64_encode(...) (utf-8), and each script creates one client and passes it into the helpers. They auto-load .env if python-dotenv is installed — find_dotenv() walks up from cwd so the file works at the repo root, the project root, or inside references/. Without python-dotenv, the scripts fall back to plain environment variables (export ATAI_API_KEY=... ATAI_API_ENDPOINT=...).
File Layout
skills/atai-newton-fusion-model/
├── SKILL.md ← this file
├── references/
│ ├── _common.py ← shared helpers on the official archetypeai client
│ ├── text_query.py ← 3 text patterns (plain, JSON output, .txt flow-log attachment)
│ ├── image_query.py ← 4 image patterns (file_ids, base64, JSON extraction, multi-image)
│ ├── video_query.py ← 3 video patterns (mp4 direct, max_frames tradeoff, frame list + query_metadata)
│ ├── csv_vs_txt_proof.py ← diagnostic: proves text/csv attachments aren't read, text/plain are
│ ├── requirements.txt ← runtime deps (pip install -r requirements.txt)
│ ├── .env.example ← copy to .env and fill in
│ └── sample_assets/
│ ├── wind-turbines.png ← AI-generated photo (default for image_query.py)
│ ├── assembly_before.png / assembly_after.png ← frames from the clip below (multi-image demo)
│ ├── 1_pass_2_pass_3_pass_B.mp4 ← assembly-inspection clip (default for video_query.py)
│ └── README.md ← provenance of the samples
└── tests/
└── test_references.py ← network-free unit tests (python -m unittest)
