expr-engine

# Writing an extremers engine This is the canonical copy of the expr-engine skill (studio/skills/expr-engine). The authorities behind it are the monorepo's `engine/PROFILE.md` (the contract) and `lang/SPEC.md` (the language); the worked example is the Race project (`api.project.create('race', { template: 'race' })`, vendored from `vendor/examples/race/`). Where the specs describe more than the compiler builds, this skill says what is built today. The compiler decides: if this skill and `api.engine.check()` disagree, believe the compiler and fix the skill. An engine is four files in the project's `engine/` directory: `engine.expr`, `state.schema.json`, and optionally `view.schema.json`, `event.schema.json`; `samples.json` holds the sample runs. **Words.** An engine is *evaluated*, once *per event*. Never call it a "step", a "tick" or a "loop": nothing runs on a clock. ## 1. What you deliver | File | Contents | |---|---| | `engine.expr` | **One expression.** It is a pure function `(state, event, timestamp, gps) → {state, view?, wake?}`, evaluated once for each event. It returns the whole new state, the view if the engine has one, and when it next wants a `wake` event. | | `state.schema.json` | The state's shape. | | `view.schema.json` | Optional. What clients receive. With it, the output **must** have `view`. Without it, the output must **not** have `view`, and clients receive the state. | | `event.schema.json` | Optional. The author's own events, which clients send. | There is nothing else: no `init`, no `step`, no `view.expr`, no `timer.expr`, no tick, no `input.tag`. ## 2. Inputs, events and the output **Inputs** | Input | Type | Meaning | |---|---|---| | `state` | the state schema, **nullable** | The last accepted state. **`nil` on the first event**, and until an evaluation is accepted. | | `event` | variants: the built-in `gps` and `wake`, plus yours | The event being handled. Exactly one field is present. | | `timestamp` | int, epoch ms | Device time when the event reached the engine. | | `gps` | list of `Fix`, up to 20,480 (`maxItems` in the interface: `engine_profile::GPS_MAX_ITEMS`, the C6's track ring in flash), oldest first | Every fix of the run since boot. Its `t` strictly increases, and every `t` is at or before `timestamp`. The studio's sim holds as many as the C6, the newest 20,480 (once full the C6 drops its oldest 128 at a time, a flash sector; the sim one at a time), so look-backs behave as on the device. | `Fix` is `{t: int (epoch ms), lat: float, lon: float, alt?: float}`: degrees, south and west negative; `alt` in metres, absent when unknown. **The track holds positions only**: no speed and no heading of past fixes. A past course or speed is derived from positions (`bearing`/`distance` between fixes, over their `t`), never read. **Events.** An evaluation happens only when an event arrives. Events are asynchronous: GPS updates come at whatever rate the receiver gives (or never); client events whenever a client sends one. - `gps`, built in: `{fix?: Fix, velocity?: {speed, heading}}`. Either part may be absent: `{gps: {}}` is a valid event. `speed` is speed over ground in **knots**; `heading` is course over ground, degrees true, 0 ≤ x < 360. When `fix` is present it is also the last item of `gps`. `event.gps.velocity` is the receiver's **current** velocity only, and only in this event: nothing keeps past velocities unless you do. To average past velocities, keep them in a bounded list in the state (section 3, "Lists"; section 5, "A list in the state: Tune"), keep a running aggregate (a sum and a count, an exponential average; a heading's average moves by a fraction of `angle_diff(v.heading, avg)` and is wrapped back into 0..360 by hand, never averaged as plain numbers), or derive it from the track's positions (section 5, "A mean direction"). - `wake`, built in: `{}`. The time you asked for has come. It carries no data, and it can arrive when you no longer want it (the state moved on since you asked): decide from the state, and leave the state alone if there is nothing to do. - Your events: the variants of `event.schema.json`, e.g. `bump_seq {seconds, tapped_at}`, `race_finish {}`. You can't name one `gps` or `wake`. The host validates a client event against the schema (ranges, enums, lengths) before evaluating; one that fails is dropped. **Output**, a record with exactly these fields: | Field | Required | Meaning | |---|---|---| | `state` | yes | The whole new state: the next evaluation's `state`. | | `view` | exactly when `view.schema.json` exists | What clients receive, computed from the new state. | | `wake` | no | Epoch ms on the device clock: when the host should next deliver `wake`. `nil` or absent means no timer. | Any other field is a compile error: `E-FIELD` "the output has no field `debug`" / "its fields are: state, wake". **The wake timer.** The host has one timer per engine. After each accepted evaluation it sets the timer to the output's `wake`, or clears it when `wake` is absent. A time already past fires at once. If you want something to happen at a time (a start gun, a reminder, a periodic beep), compute that time and return it as `wake` on every evaluation while you still want it. **A failed evaluation** (a run-time error) keeps the old state and drops the event. If no evaluation is ever accepted, `state` stays `nil`, so **an engine that fails when `state` is nil never starts**. **Clients format.** What clients receive holds values, not text: `pace_s_per_km: 323`, never `"5:23"`. Put units in field names (`_ms`, `_m`, `_kn`). ## 3. Schemas The schemas use the wire subset of JSON Schema, plus bounds. The compiler rejects violations as `E-SCHEMA` with the path. | Kind | Write | Notes | |---|---|---| | integer | `{"type": "integer", "format": "int64"}` | Give a `format` (`int32`, `int64`). `minimum`/`maximum` become a static range the checker enforces (`E-RANGE`). Times and durations are int64 epoch ms / ms. `uint64` is rejected. | | float | `{"type": "number"}` | | | bool | `{"type": "boolean"}` | | | string | `{"type": "string", "maxLength": 8}` | `maxLength` is required ("a string needs `maxLength`"). | | enum | `{"enum": ["port", "stbd"]}` | | | record | `{"type": "object", "required": [...], "properties": {...}}` | A field not in `required` is optional, and optional means **absent**: there are no null values. | | list | `{"type": "array", "maxItems": 5, "items": ...}` | `maxItems` is required ("a list needs `maxItems`"). It is also what the list costs: see "Lists" below. | | variants | `oneOf` (below) | | **Variants** say "exactly one of these". Each alternative names one variant: it has exactly one property, lists it in `required`, and that property's schema is an object. From Race's state: ```json "phase": { "oneOf": [ {"properties": {"idle": {"type": "object"}}, "required": ["idle"]}, {"properties": {"countdown": {"type": "object", "required": ["gun"], "properties": {"gun": {"type": "integer", "format": "int64"}}}}, "required": ["countdown"]}, {"properties": {"racing": {"type": "object", "required": ["gun"], "properties": {"gun": {"type": "integer", "format": "int64"}}}}, "required": ["racing"]} ] } ``` In JSON a value is `{"countdown": {"gun": 1790000300000}}`. An alternative that isn't a named record (`{"type": "integer"}`, or one tagged by a `"kind"` string) is `E-SCHEMA`: "every alternative is a record that names its variant". `oneOf` isn't allowed inside list items. `event.schema.json` is a top-level `oneOf` of variants, one per event, as in Race's (section 10). **Model "which of several" with variants, never with a tag string.** Use an enum only for a value that an event carries. **Lists (as built).** A list can go in the state and in the view. Its value can be a list literal (`[]`, `[{t: timestamp, v: x}]`), a list passed on (`s.samples`), or a list the program builds with `map`, `filter`, `concat`, a slice `xs[a:b]`, `take(xs, n)`, `reverse`, a range `a..b`, `flatten`, `sortBy`/`sort` or `uniq` (section 7, "Reordering, deduplicating and flattening", for what each holds). - **The bounded-list idiom.** Append and keep the newest N: `concat(state?.xs ?? [], [x])[-N:]`, with the field's `maxItems` at least N. Without the slice it is `E-OUTPUT-BOUND` "the output field `state.samples` holds at most 64 items, but this can have 65" / "keep the last 64: `(…)[-64:]`". To keep a time window, filter by time and still bound it: `filter(concat(s.xs, [x]), .t >= timestamp - 30000)[-64:]`. Once the list is capped `len` stops growing, so keep a separate count if you need one. - **What a built list costs.** A list literal, and a list the program builds where a value is needed (an output field, a record field, a branch, a `??`, a `let` used as a value), is *held*: a block of `bound × item leaves` locals, reserved for the **whole evaluation**. The bound is the most items the compiler can prove: `xs[-N:]` has at most N items, `concat` the sum of its parts, `filter` and `map` their input's. An item's leaves are one per scalar field (plus one for each optional record in it). The block counts against the program's locals, and the C6 allows **256 locals** for everything. `api.engine.compile()` reports `memory.held_slots`, the locals that hold lists. **The compile does not stop you at 256**: it is the host that refuses the bundle, `api.sim.start()` (and the box at install) with "the engine needs more than the C6 has: locals". So check `memory.locals ≤ 256` after every change to a list's `maxItems`. (With `api.engine.compile({ limitBytes })` the compile fails with `E-LIMIT`, and its help names the held slots.) Beyond 32,768 held slots the compile fails with `E-HOLD`. Tune (section 5) keeps 64 samples of 3 fields: its lists hold 198 of its 225 locals. A list consumed by another list builtin (`sum(map(xs, .v))`, `last(filter(…))`) is never held and costs nothing, except what a sort, `uniq` or `median` holds (section 7). - **Hold a slice of `gps`, never `gps` itself.** A held `filter(gps, …)` or `map(gps, …)` has a bound of 20,480 (the schema can't express the gap between fixes), which can never fit. Slice by count first: `map(gps[-11:], ({lat: .lat, lon: .lon}))`. The same goes for a sort, `uniq` or `median` of `gps` that holds its input: `median(map(gps[-60:], …))` holds 60, `median(map(gps, …))` 20,480. - **A state list round-trips on the device** (engine/PROFILE.md §3, "State lists"). After each accepted evaluation the host copies each list in the state into a region of its own, `maxItems × item leaves` slots. It serves that region to the next evaluation as `state`'s list. A rejected evaluation keeps the last state and its lists. Nothing survives a reboot. The C6 has **1,024 slots for up to 4 state lists** in all. A bundle over either limit is refused at install (`OverLimit`: state-list slots, or state lists). `api.engine.layout().stateLists` lists the state's lists with their `maxItems` and `itemLeaves`. In practice the 256 locals bind first: a list the program builds must also fit in them. - **A list in the view** is encoded in full on every publish, and the published value must fit 256 bytes. Keep view lists short. ## 4. Variants and nil **Test variants and events by presence.** Never compare strings. - Dispatch on the event: `event.gps != nil ? … : event.wake != nil ? … : event.bump_seq != nil ? … : s`. - Read a variant's data: `s.phase.countdown?.gun` (nil unless counting down). After `s.phase.countdown != nil ?`, read `s.phase.countdown.gun` directly. - `event.tag` is `E-NAME` "`event` has no field `tag`", and `input` is `E-NAME` "there's no input or name `input`". **Exactly one variant in an output.** Wherever the output schema has variants, the checker must prove exactly one is present (`E-VARIANT`). It accepts: - a record literal with exactly one variant, whose value can't be nil: `{idle: {}}`, `{racing: {gun: s.phase.countdown.gun}}`; - a value whose type already has the variants: `s.phase`, the whole `s`, or `state ?? {phase: {idle: {}}, line: {}}`; - a conditional or `??` whose every branch is accepted, and `let` names bound to accepted values. It rejects, with these messages: | You wrote | `E-VARIANT` says | |---|---| | `{a: {}, b: {at: timestamp}}` | "must have exactly one of `a`, `b`, `c`, but this sets 2 of them: `a` and `b`" | | `{}` | "… but this sets none of them" | | `{b: b}` with `b` nullable | "… but the variant `b` here may be nil" / "choose the variant with a condition, e.g. `b != nil ? {b: b} : {a: …}`" | The fix for the last one compiles: `let b = state?.phase?.b; {state: {n: 0, phase: b != nil ? {b: b} : {a: {}}}}`. **What the checker insists on about nil.** It only stops a possibly-nil value going where nil can't go: - a required output field: `E-TYPE` "the output field `state.n` can't be nil, but this is an integer (0..10) or nil";