protocol

name	protocol
description	Ingest protocol reference between the ESP32-C3 device and the helmlog server — schema, versioning, idempotency, endpoint contract. TRIGGER when adding fields to the sample payload, bumping the schema version, implementing the helmlog-side endpoint, or debugging an ingest round-trip. DO NOT trigger for BNO085 register-level questions (use /calibrate or /domain).

Ingest protocol (v1)

The wire contract between imu4helmlog (device) and helmlog (server). Owned jointly — changes to this skill must be mirrored in the helmlog-side route handler and the helmlog schema migration.

Endpoint

POST /api/v1/imu/ingest
Content-Type: application/json
Content-Encoding: gzip
X-Device-Id: <hex>         # same as body.device_id; for log routing before parse
X-Boot-Id: <hex>

The URL is versioned via path (v1). Additive changes (new optional fields) stay on v1. Breaking changes get v2 and both endpoints must coexist for at least one deploy cycle so old devices don't brick during an upgrade.

Body schema

{
  "schema": 1,                 // int; bump only for breaking changes
  "device_id": "c3a1b2...",    // 64-bit device id from NVS, hex
  "boot_id": "9f...",          // random at boot, 64-bit hex
  "seq": 1234,                 // uint, monotonic per (device_id, boot_id)
  "sent_at_us": 15000000,      // device monotonic micros when POST fired
  "wall_clock_offset_ms": 1733000000000,  // epoch ms at boot_us=0, 0 if pre_sntp
  "pre_sntp": false,           // true if SNTP hasn't completed this boot
  "mount_quat": [w, x, y, z],  // sensor→boat frame, set at install
  "samples": [
    {
      "t_us": 12345678,        // boot-monotonic micros, MUST be increasing
      "quat": [w, x, y, z],    // game rotation vector, sensor→world, unit
      "lin_accel": [ax, ay, az],  // m/s², sensor frame
      "gyro":      [gx, gy, gz],  // rad/s, sensor frame
      "accel":     [ax, ay, az],  // m/s², sensor frame, raw (includes gravity)
      "acc_status": 3           // 0=unreliable, 1=low, 2=med, 3=high
    }
  ]
}

Invariants the server must reject on:

• schema not in supported set → 400
• samples[i].t_us <= samples[i-1].t_us → 400 (monotonicity is the whole point of using boot micros instead of wall clock)
• len(samples) > 500 → 413 (1s batch at 400Hz absolute ceiling)
• quaternion norm > 1.01 or < 0.99 → 400 (sensor sent garbage)
• mount_quat all zeros → 422 (device is pre-calibration, should not be uploading; device-side packetizer should already refuse this)

Idempotency and dedupe

(device_id, boot_id, seq) is the idempotency key. The server:

1. Looks up MAX(seq) WHERE device_id=? AND boot_id=?
2. If incoming.seq <= max: still 200, but response {"deduped": true} and does not re-store samples
3. Else: insert all samples, bump max

Boot IDs change every boot, and seq restarts at 0. That's fine — (boot_id, seq) uniquely identifies a batch forever. The device never reuses a boot_id.

Response

{
  "accepted": 100,        // number of samples actually stored
  "deduped": false,
  "last_seq": 1234,       // highest seq the server has for this boot_id
  "server_time_ms": 1733000001234   // for optional clock-drift telemetry
}

HTTP codes:

Code	Meaning	Device action
200	Accepted or deduped	Advance upload pointer, drop from ringbuf
400	Malformed (schema/monotonicity/quat norm)	Log loudly, drop batch — retrying won't help
413	Batch too big	Split and retry (should never happen in practice)
422	Pre-cal or other semantic reject	Device should not be uploading — halt uploader
429	Rate limited	Back off per Retry-After header
5xx	Server or network	Retry with exponential backoff, max 30 s, keep in ringbuf

The device never gives up on 5xx. Ringbuf overflow drops oldest batches, not newest.

Helmlog-side storage (target shape)

A new migration in helmlog/src/helmlog/storage.py adds:

CREATE TABLE imu_samples (
    device_id TEXT NOT NULL,
    boot_id   TEXT NOT NULL,
    seq       INTEGER NOT NULL,    -- batch seq, not per-sample
    t_utc_ms  INTEGER NOT NULL,    -- reconstructed: wall_clock_offset_ms + t_us/1000
    pre_sntp  INTEGER NOT NULL,    -- 1 if t_utc_ms is unreliable
    qw REAL, qx REAL, qy REAL, qz REAL,     -- boat frame (mount_quat applied server-side)
    lax REAL, lay REAL, laz REAL,            -- linear accel, boat frame
    gx REAL, gy REAL, gz REAL,               -- gyro, boat frame
    acc_status INTEGER
);
CREATE INDEX imu_samples_time ON imu_samples(t_utc_ms);
CREATE UNIQUE INDEX imu_samples_seq ON imu_samples(device_id, boot_id, seq, t_utc_ms);

Frame question — where does mount_quat get applied, device or server?

On the server. Reasons:

• Keeps the wire format sensor-native — a bad mount_quat value doesn't silently corrupt stored data; we can re-derive with a corrected value
• Lets us replay old captures under a new mount calibration
• Device CPU is free but round-off compounds if we rotate on device and again on server

mount_quat ships in every batch header for exactly this reason — the server applies it in the route handler, before insert.

Protocol versioning rules

1. Additive (no schema bump): new optional fields in the batch or sample object. Device may send, server must ignore unknown fields
2. Breaking (bump schema and endpoint path): removing fields, changing semantics, changing types. Server supports old and new for one release
3. schema is checked first — mismatched schema is a 400, not a 422, because the body can't be interpreted far enough to know what's in it

Alternative routes considered (and why not)

• Signal K deltas to the local SK server: architecturally cleaner (matches helmlog's "SK is primary" principle). Rejected for v1 because it couples device liveness to SK server liveness and constrains the schema to SK paths; seastate-specific fields don't have standard paths. Revisit in v2 if the seastate analysis stabilizes and fits SK shape
• MessagePack / CBOR: ~40% smaller on the wire. Rejected for v1 because JSON is debuggable with curl and jq, and gzip closes most of the gap. Revisit if bandwidth becomes a problem

Known gotchas

• wall_clock_offset_ms is set once per boot, not continuously. SNTP drift during a 12-hour race is negligible compared to sample-timing jitter
• gzip Content-Encoding is end-to-end — do not let a reverse proxy re-compress. FastAPI + uvicorn pass it through unchanged
• t_us rolls over at ~71 minutes (uint32 micros) — packetizer widens to uint64 via a rollover-aware counter; the wire format uses uint64 to avoid any ambiguity. Don't let anyone "optimize" this back to uint32