atai-newton-fusion-model

name

atai-newton-fusion-model

description

Call Newton's C 2.6 fusion model (`Newton::c2_6_8b_fp8_260424d7a55d5e`) on the prod `/query` endpoint with text, image, and video inputs in a single stateless request — no session, no batch job, no SSE plumbing. Use this skill when the user wants per-call multimodal reasoning (describe an image, summarize a clip, classify a state from sensor + visual context) without managing session lifecycle, or when they need the Newton C checkpoint that reasons over video frames via `/query`. Covers the request shape per modality, the two image-attachment paths (file_ids vs base64), multi-image mode, both video paths (`.mp4` + `max_frames`, and client-sampled frames + `query_metadata`), generation parameters, JSON-output prompting, latency budgets, and the C 2.6 identifier gotcha. Do NOT use for streaming / session-based activity monitoring, large multi-file batch jobs, or time-series embedding (KNN / anomaly) classification.

Newton C 2.6 Fusion Model — Multimodal /query in One Hop

Single stateless POST to /query for text, image, or video reasoning against the C 2.6 fusion checkpoint. No Lens session, no batch pipeline, no SSE. One call → one response.

When to Apply

• User wants to describe / classify / extract structure from an image with one API call
• User wants the same for a short video in a single /query call
• User wants stateless multimodal reasoning (no per-session warmup, no orphan cleanup)
• User wants JSON output from a multimodal input (the prompt is the schema)
• User is building a serverless / per-request handler where a streaming Lens session would be the wrong shape

Do not use this skill when:

• The workload is live, streaming video → text from a long-running camera feed — a session-based activity monitor is the right shape, not stateless /query
• The workload is a large multi-file batch job
• The task is time-series classification via embeddings (KNN / anomaly detection)

The Model Identifier

Newton::c2_6_8b_fp8_260424d7a55d5e

The full Newton:: prefix and the _fp8_ segment are required. Variants that omit either are rejected with 400 invalid_model_version:

c2_6_8b_fp8_260424d7a55d5e            ❌ missing Newton:: prefix
Newton::c2_6_8b_260424d7a55d5e        ❌ missing _fp8_
c2_6_8b_260424d7a55d5e                ❌ both missing

This was probed end-to-end against https://api.u1.archetypeai.app/v0.5/query on prod.

Endpoint

POST {ATAI_API_ENDPOINT}/v0.5/query
Authorization: Bearer <API_KEY>
Content-Type: application/json

Both ATAI_API_KEY and ATAI_API_ENDPOINT are required — there is no default endpoint, so a wrong-deployment mistake fails loudly at startup instead of silently at query time. Prod is https://api.u1.archetypeai.app/v0.5. The same endpoint serves all three modalities below — what changes is the request body.

Wire Shapes

Text + JSON

{
  "query": "Classify the operational state. Respond with ONLY the JSON object.",
  "instruction_prompt": "You output {\"state\": \"...\", \"confidence\": <0..1>}.",
  "file_ids": [],
  "model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
  "max_new_tokens": 300
}

• Put the system turn in instruction_prompt. That is the only field C 2.6 honors — a directive sent in the legacy system_prompt field alone is silently ignored (verified against Newton::c2_6_8b_fp8_260424d7a55d5e: the same directive is obeyed in instruction_prompt and dropped in system_prompt), so omit system_prompt entirely.
• JSON output: put the schema in instruction_prompt, the input in query, and explicitly tell the model not to wrap in markdown fences. The model otherwise often returns json ... .

For reasoning over an attached text file: upload it as text/plain (a .txt file) via /v0.5/files, then put the filename in file_ids. The model then sees the file contents in its prompt context. Critical: this only works for text/plain. A file uploaded as text/csv (a .csv) is accepted and stored byte-identically, but its contents never reach the model on /query — it silently confabulates from priors (see the gotcha below and csv_vs_txt_proof.py). application/json uploads are rejected outright. So put tabular data in a .txt, or inline it in query.

See text_query.py for runnable demos of all three patterns. Its examples are framed as smart-home WiFi flow analysis; the attached flow log in example 3 is real data from the GHOST-IoT dataset (Anagnostopoulos et al., Sensors 2020, DOI:10.3390/s20226600) — see the script's docstring for full attribution.

Image

Two attachment paths, both reach the same fusion model — pick by use case — plus a flag for sending several images at once:

(a) Upload + reference by filename in file_ids — best when the same image is reused across queries (one upload, many /query calls):

curl -X POST $ATAI_API_ENDPOINT/v0.5/files \
  -H "Authorization: Bearer $ATAI_API_KEY" \
  -F "file=@dashboard.png;type=image/png"

{
  "query": "Describe this image in three bullets.",
  "instruction_prompt": "...",
  "file_ids": ["dashboard.png"],
  "model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
  "max_new_tokens": 300
}

(b) Inline base64 in a data.base64_img event — best for one-shot calls where the upload roundtrip is wasted:

{
  "query": "What is in this image? One sentence.",
  "instruction_prompt": "...",
  "file_ids": [],
  "model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
  "max_new_tokens": 200,
  "events": [
    {
      "type": "data.base64_img",
      "event_data": {
        "contents": "<base64 string>",
        "mime_type": "image/png"
      }
    }
  ]
}

(c) Multiple images — set multi_image: true for multi-image mode: each attachment is an independent image (before/after, multi-view, "is the same person in these images?"), not frames of a video. Platform limit: 16 images max per request. Without multi_image: true, an image list means "frames of one video" and requires query_metadata (see the Video section); lacking both, the request fails with 400 query_failed:

{
  "query": "Image 1 is the bench BEFORE the task; image 2 is DURING. What changed?",
  "instruction_prompt": "...",
  "file_ids": ["assembly_before.png", "assembly_after.png"],
  "model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
  "max_new_tokens": 400,
  "multi_image": true
}

See image_query.py for runnable demos of all three paths.

Video

C 2.6 reasons over video on /query two ways. This distinguishes it from the C 2.4 / 2.5 text checkpoints, which accept an .mp4 but ignore the frames and reply "I can't see videos."

(a) .mp4 + max_frames — upload the clip, and GPQ decodes and uniformly samples it server-side before the model sees the frames. No client-side video tooling needed:

curl -X POST $ATAI_API_ENDPOINT/v0.5/files \
  -H "Authorization: Bearer $ATAI_API_KEY" \
  -F "file=@clip.mp4;type=video/mp4"

{
  "query": "Evaluate the assembly steps shown in the attached video.",
  "instruction_prompt": "<task / inspection prompt>",
  "file_ids": ["clip.mp4"],
  "model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
  "max_new_tokens": 500,
  "max_frames": 32
}

(b) Client-sampled frames + query_metadata — send the frames yourself (one data.base64_img event each, or image file_ids), set multi_image: false, and include a query_metadata block with the video-metadata triple. The query_metadata block is what makes this work — without it the request fails with 400 query_failed:

{
  "query": "This is a short video. Describe the sequence of actions in order.",
  "instruction_prompt": "...",
  "model": "Newton::c2_6_8b_fp8_260424d7a55d5e",
  "multi_image": false,
  "max_new_tokens": 150,
  "events": [
    {"type": "data.base64_img", "event_data": {"contents": "<frame_0 base64>"}},
    {"type": "data.base64_img", "event_data": {"contents": "<frame_1 base64>"}}
  ],
  "query_metadata": {"raw_fps": 1.0, "frames_indices": [0, 1], "total_num_frames": 2}
}

Both shapes verified live (the file_ids variant of (b) works too). Notes:

• max_frames — frames GPQ samples uniformly from the video before feeding the model. Default 32, platform max 64. Increase for longer / higher-detail clips; decrease to cap latency. Values above 64 fail silently: the request returns 200, but the video is dropped and the model confabulates a scene from priors (verified: max_frames: 80 on the assembly clip described "a person holding a smartphone"). Stay at ≤ 64.
• Timeout — set the client request timeout to at least 600s. Video latency scales with max_frames and clip length; default 30–60s timeouts will trip on non-trivial clips.
• .mp4-direct on C 2.6 is a recent platform change. Earlier the .mp4 path silently dropped the frames for every model (a known GPQ bug), so older notes say "video doesn't work on /query." On C 2.6 today it does, via max_frames.
• multi_image selects between video and multi-image mode. multi_image: false (with query_metadata) = the attached images are frames of one video. multi_image: true = multi-image mode — the images are independent (before/after, multi-view), and query_metadata is not needed.

See video_query.py for runnable demos of both paths (mp4 direct, a max_frames tradeoff, and a frame-list demo) using a worker-assembly PASS/FAIL inspection prompt.

Generation Parameters

All C-model generations via /query accept sampling controls in the request body (verified accepted + honored on prod with C 2.6):

Param	Default	Notes
max_new_tokens	256	Hard ceiling on generated tokens (~4 chars each) — stops mid-sentence at the cap
temperature	—	0.0 = greedy/deterministic-ish; 0.7–1.0 for creative output. Capped at 2.0
do_sample	true	Master switch: false forces greedy decoding (temperature → 0, top_p/top_k stripped) — use for reproducibility
repetition_penalty	1.0	Multiplicative logit penalty on already-emitted tokens; try 1.1–1.3 if the model loops on a phrase or repeats numbers
presence_penalty	0.0	Additive penalty on any token that has appeared at least once (frequency-independent). Range ≈ −2..+2. C 2.6 only
top_p	0.8	Nucleus sampling — keep the smallest token set with cumulative probability ≥ top_p
top_k	—	Hard cap on candidate tokens per step (applied before top_p); 0 disables

For "extract X"-style tasks, do_sample: false (or temperature: 0.0) plus a tight max_new_tokens is the right starting point.

Reproducibility caveat (verified): the sampling knobs are honored — on the same prompt, 5 runs gave 2 distinct outputs greedy (do_sample: false), 3 distinct at defaults, 5 distinct at temperature: 1.9 — but greedy is not bit-deterministic on this deployment (fp8 numerics / batching sit outside the decoder), contrary to the platform note that do_sample: false output "is deterministic given the prompt". On borderline judgments the residual wobble is enough to flip the answer: the same greedy assembly-inspection body returned all-PASS on one run and two FAILs on another. If a verdict matters, aggregate over a few calls rather than trusting one.

Latency Budgets

Observed against prod with the sample assets in references/sample_assets/:

Modality	Body	p50	Notes
Text only	{"query": "...", "file_ids": []}	~500 ms	Confirmed during model-identifier probing
Text + JSON output	instruction prompt has schema	~600 ms	Same shape, slightly longer with verbose schema
Image via file_ids	one 1.9 MB PNG	~1.3–1.9 s	observed (wind-turbines.png)
Image via inline base64	same	~0.8–2.5 s	base64 inflates the request; scales with image size
Two images, multi_image: true	2 × ~400 KB PNG	~2.9 s	before/after comparison (assembly frames)
Video, mp4 + max_frames=32	~31 s assembly clip	~2–6 s	server-side decode/sample
Video, 2 frames + query_metadata	2 × ~400 KB PNG	~1.9 s	client-sampled frame-list path

Numbers are smoke-test order-of-magnitude. Production data may be 5–10× heavier and proportionally slower.

Response Shape

{
  "query_id": "...",
  "status": "completed",
  "inference_time_sec": 0.42,
  "query_response_time_sec": 0.49,
  "response": {
    "response": ["<the model output as a string>"]
  }
}

Walk payload["response"]["response"][0] as the canonical extraction path. The reference scripts' extract_text() helper handles every shape variant we've seen.

Common Pitfalls

• Wrong model identifier. c2_6_8b_fp8_260424d7a55d5e without Newton:: returns 400 invalid_model_version. So does the variant without _fp8_. Always use the full Newton::c2_6_8b_fp8_260424d7a55d5e.
• JSON wrapped in markdown fences. The model sometimes returns json ... even when explicitly told not to. Either parse defensively (text.strip().removeprefix("```json").removesuffix("```")) or restate the no-fences rule in the user query as well as instruction_prompt.
• Video timeout. Default requests.post timeout of 30–60s will fail on anything but trivial clips. Set timeout ≥ 600s.
• Multiple images without multi_image: true need the query_metadata triple. A list of images defaults to "frames of one video" — but that path requires query_metadata: {"raw_fps": ..., "frames_indices": [...], "total_num_frames": N} in the body; without it the request fails with 400 query_failed (the error doesn't hint at the missing field). For independent images (before/after, multi-view), set multi_image: true instead — max 16 images, no query_metadata needed.
• max_frames above 64 fails silently. The platform max is 64. max_frames: 80 still returns 200, but the video never reaches the model — it confabulates a scene from priors (same failure mode as the CSV gotcha below). Keep max_frames ≤ 64.
• "Video doesn't work on /query" is stale. Older notes say C 2.6 (and 2.4/2.5) can't see video. That was a GPQ bug that silently dropped .mp4 frames before any model. C 2.6 reads video now via .mp4 + max_frames — verified. The 2.4/2.5 text checkpoints still ignore frames.
• file_ids takes the filename, not the fil_... uid. On upload the API returns both {"file_id": "<filename>", "file_uid": "fil_..."}. /query resolves file_ids by the filename (the file_id value, e.g. "dashboard.png"), not the fil_... file_uid. Passing the file_uid fails with a misleading 400 unsupported_file_type ("Found unsupported file type with file: fil_…") — it looks like a content-type problem but is really the wrong identifier. Verified: file_ids=["wind-turbines.png"] → 200; file_ids=["fil_…"] → 400.
• Text attachments must be text/plain, not text/csv. A .csv upload (text/csv) is accepted, stored byte-for-byte, and returns status: completed — but its contents never reach the model on /query; the model silently confabulates from priors (e.g. answers "TCP"/"port 80" for an all-SSH file). The identical bytes uploaded as .txt (text/plain) are read faithfully. There is no warning — the failure is silent, so you can only catch it by validating the answer. Upload tabular/text data as .txt, or inline it in query. Verify yourself with csv_vs_txt_proof.py (.txt reads 5/5, .csv 0/5 on identical content). application/json uploads are rejected with 400 invalid_file_type.
• Mixed-modality calls. You can attach both an image and a video file_id, but the fusion model's attention is finite. Stick to one modality per call for predictable results.

Local Setup

# In a virtualenv (system pythons are often PEP 668 "externally managed"):
python3 -m venv .venv && source .venv/bin/activate
pip install -r skills/atai-newton-fusion-model/references/requirements.txt

# Drop a .env at the repo root (or anywhere up the tree from the run dir).
# BOTH variables are required — there is no default endpoint:
cat > .env <<EOF
ATAI_API_KEY=sk_...
ATAI_API_ENDPOINT=https://api.u1.archetypeai.app/v0.5
EOF

python skills/atai-newton-fusion-model/references/text_query.py
python skills/atai-newton-fusion-model/references/image_query.py
python skills/atai-newton-fusion-model/references/video_query.py

The scripts are built on the official Archetype AI python client (pip install archetypeai): uploads go through client.files.local.upload(...), base64 encoding through archetypeai.utils.base64_encode(...) (utf-8), and each script creates one client and passes it into the helpers. They auto-load .env if python-dotenv is installed — find_dotenv() walks up from cwd so the file works at the repo root, the project root, or inside references/. Without python-dotenv, the scripts fall back to plain environment variables (export ATAI_API_KEY=... ATAI_API_ENDPOINT=...).

File Layout

skills/atai-newton-fusion-model/
├── SKILL.md                  ← this file
├── references/
│   ├── _common.py            ← shared helpers on the official archetypeai client
│   ├── text_query.py         ← 3 text patterns (plain, JSON output, .txt flow-log attachment)
│   ├── image_query.py        ← 4 image patterns (file_ids, base64, JSON extraction, multi-image)
│   ├── video_query.py        ← 3 video patterns (mp4 direct, max_frames tradeoff, frame list + query_metadata)
│   ├── csv_vs_txt_proof.py   ← diagnostic: proves text/csv attachments aren't read, text/plain are
│   ├── requirements.txt      ← runtime deps (pip install -r requirements.txt)
│   ├── .env.example          ← copy to .env and fill in
│   └── sample_assets/
│       ├── wind-turbines.png ← AI-generated photo (default for image_query.py)
│       ├── assembly_before.png / assembly_after.png ← frames from the clip below (multi-image demo)
│       ├── 1_pass_2_pass_3_pass_B.mp4 ← assembly-inspection clip (default for video_query.py)
│       └── README.md         ← provenance of the samples
└── tests/
    └── test_references.py    ← network-free unit tests (python -m unittest)