protocol

# Ingest protocol (v1) The wire contract between `imu4helmlog` (device) and `helmlog` (server). Owned jointly — changes to this skill must be mirrored in the helmlog-side route handler and the helmlog schema migration. ## Endpoint ``` POST /api/v1/imu/ingest Content-Type: application/json Content-Encoding: gzip X-Device-Id: <hex> # same as body.device_id; for log routing before parse X-Boot-Id: <hex> ``` The URL is versioned via path (`v1`). Additive changes (new optional fields) stay on `v1`. Breaking changes get `v2` and both endpoints must coexist for at least one deploy cycle so old devices don't brick during an upgrade. ## Body schema ```jsonc { "schema": 1, // int; bump only for breaking changes "device_id": "c3a1b2...", // 64-bit device id from NVS, hex "boot_id": "9f...", // random at boot, 64-bit hex "seq": 1234, // uint, monotonic per (device_id, boot_id) "sent_at_us": 15000000, // device monotonic micros when POST fired "wall_clock_offset_ms": 1733000000000, // epoch ms at boot_us=0, 0 if pre_sntp "pre_sntp": false, // true if SNTP hasn't completed this boot "mount_quat": [w, x, y, z], // sensor→boat frame, set at install "samples": [ { "t_us": 12345678, // boot-monotonic micros, MUST be increasing "quat": [w, x, y, z], // game rotation vector, sensor→world, unit "lin_accel": [ax, ay, az], // m/s², sensor frame "gyro": [gx, gy, gz], // rad/s, sensor frame "accel": [ax, ay, az], // m/s², sensor frame, raw (includes gravity) "acc_status": 3 // 0=unreliable, 1=low, 2=med, 3=high } ] } ``` **Invariants the server must reject on:** - `schema` not in supported set → 400 - `samples[i].t_us <= samples[i-1].t_us` → 400 (monotonicity is the whole point of using boot micros instead of wall clock) - `len(samples) > 500` → 413 (1s batch at 400Hz absolute ceiling) - quaternion norm > 1.01 or < 0.99 → 400 (sensor sent garbage) - `mount_quat` all zeros → 422 (device is pre-calibration, should not be uploading; device-side packetizer should already refuse this) ## Idempotency and dedupe `(device_id, boot_id, seq)` is the idempotency key. The server: 1. Looks up `MAX(seq) WHERE device_id=? AND boot_id=?` 2. If `incoming.seq <= max`: still 200, but response `{"deduped": true}` and **does not re-store samples** 3. Else: insert all samples, bump max **Boot IDs change** every boot, and `seq` restarts at 0. That's fine — `(boot_id, seq)` uniquely identifies a batch forever. The device never reuses a boot_id. ## Response ```jsonc { "accepted": 100, // number of samples actually stored "deduped": false, "last_seq": 1234, // highest seq the server has for this boot_id "server_time_ms": 1733000001234 // for optional clock-drift telemetry } ``` HTTP codes: | Code | Meaning | Device action | |---|---|---| | 200 | Accepted or deduped | Advance upload pointer, drop from ringbuf | | 400 | Malformed (schema/monotonicity/quat norm) | Log loudly, **drop** batch — retrying won't help | | 413 | Batch too big | Split and retry (should never happen in practice) | | 422 | Pre-cal or other semantic reject | Device should not be uploading — halt uploader | | 429 | Rate limited | Back off per `Retry-After` header | | 5xx | Server or network | Retry with exponential backoff, max 30 s, keep in ringbuf | The device **never gives up** on 5xx. Ringbuf overflow drops *oldest* batches, not newest. ## Helmlog-side storage (target shape) A new migration in `helmlog/src/helmlog/storage.py` adds: ```sql CREATE TABLE imu_samples ( device_id TEXT NOT NULL, boot_id TEXT NOT NULL, seq INTEGER NOT NULL, -- batch seq, not per-sample t_utc_ms INTEGER NOT NULL, -- reconstructed: wall_clock_offset_ms + t_us/1000 pre_sntp INTEGER NOT NULL, -- 1 if t_utc_ms is unreliable qw REAL, qx REAL, qy REAL, qz REAL, -- boat frame (mount_quat applied server-side) lax REAL, lay REAL, laz REAL, -- linear accel, boat frame gx REAL, gy REAL, gz REAL, -- gyro, boat frame acc_status INTEGER ); CREATE INDEX imu_samples_time ON imu_samples(t_utc_ms); CREATE UNIQUE INDEX imu_samples_seq ON imu_samples(device_id, boot_id, seq, t_utc_ms); ``` **Frame question — where does `mount_quat` get applied, device or server?** On the **server**. Reasons: - Keeps the wire format sensor-native — a bad `mount_quat` value doesn't silently corrupt stored data; we can re-derive with a corrected value - Lets us replay old captures under a new mount calibration - Device CPU is free but round-off compounds if we rotate on device and again on server `mount_quat` ships in every batch header for exactly this reason — the server applies it in the route handler, before insert. ## Protocol versioning rules 1. **Additive (no schema bump)**: new optional fields in the batch or sample object. Device may send, server must ignore unknown fields 2. **Breaking (bump `schema` and endpoint path)**: removing fields, changing semantics, changing types. Server supports old and new for one release 3. **`schema` is checked first** — mismatched schema is a 400, not a 422, because the body can't be interpreted far enough to know what's in it ## Alternative routes considered (and why not) - **Signal K deltas to the local SK server**: architecturally cleaner (matches helmlog's "SK is primary" principle). Rejected for v1 because it couples device liveness to SK server liveness and constrains the schema to SK paths; seastate-specific fields don't have standard paths. Revisit in v2 if the seastate analysis stabilizes and fits SK shape - **MessagePack / CBOR**: ~40% smaller on the wire. Rejected for v1 because JSON is debuggable with `curl` and `jq`, and gzip closes most of the gap. Revisit if bandwidth becomes a problem ## Known gotchas - **`wall_clock_offset_ms` is set once per boot**, not continuously. SNTP drift during a 12-hour race is negligible compared to sample-timing jitter - **gzip `Content-Encoding` is end-to-end** — do not let a reverse proxy re-compress. FastAPI + uvicorn pass it through unchanged - **`t_us` rolls over at ~71 minutes** (uint32 micros) — packetizer widens to uint64 via a rollover-aware counter; the wire format uses uint64 to avoid any ambiguity. Don't let anyone "optimize" this back to uint32