kimojio-json-decoder

name	kimojio-json-decoder
description	Write or update an allocation-free JSON decoder over the kimojio-json tokenizer. Use when asked to build, finish, or change code that turns JSON into a struct, an enum, or an iterator without heap allocation - especially when the answer is a state machine that would be tedious to write by hand.
license	MIT

kimojio-json decoders

kimojio-json tokenizes JSON. It reports borrowed input to a visitor. A decoder maps those events to a typed Rust value.

This skill is for direct, allocation-free decoders. It gives the agent the constraints and the reasons for them. It does not prescribe one state layout for every schema. A previous implementation is evidence, not a specification. If a task gives a type or an interface, treat that interface as a contract unless the task also asks to change it.

Start with the problem

Before writing code, identify:

• the input source and schema;
• the required public interface and input lifetime;
• whether the result is one value, a sequence, or a callback stream;
• the policy for unknown members and optional members;
• maximum sizes and nesting depth;
• error kinds, byte offsets, and completion behavior;
• the expected performance and trust boundary.

Then compare possible designs. For each design, ask:

1. What work or failure mode does it avoid?
2. Does that cost apply to this schema and interface?
3. What is the smallest representation that satisfies the contract?
4. What measurement or test would distinguish the choices?

State the reason for a choice in the module documentation or change summary. Do not copy a detailed technique from an earlier decoder without checking why it was used. A different schema may make another technique better.

Read the tokenizer

Read these files from the repository root before relying on an API detail:

• kimojio-json/src/lib.rs
• kimojio-json/src/tokenizer.rs
• kimojio-json/src/escape_string.rs
• kimojio-json/src/error.rs

Confirm names, lifetimes, return values, and error behavior in the source. Do not invent a pull-token API or assume that a callback has access to the tokenizer.

Prefer the tokenizer's visitor protocol when it fits

The tokenizer already owns the scan loop and calls a Visitor. Prefer this protocol when the decoder can consume events as they arrive. This avoids an extra token iterator and an extra dispatch layer.

This recommendation has a measured reason. In this repository, iterator overhead has dominated tokenization for some workloads. A visitor can provide the same streaming behavior while paying the suspension and output cost only for events that complete an item: return no output for other events and return an output at an item boundary. Use an iterator adapter only when the caller's interface needs one, and compare its cost with the direct visitor design.

The visitor methods return Option<Self::Output>. In the normal case, None means that scanning continues. Some returns control to drive; a later call resumes the same decoder state. This is the mechanism for streaming results. If a decoder never suspends, an uninhabited output such as core::convert::Infallible can make the impossible output path removable. Confirm the exact drive contract in the tokenizer before depending on this.

Other tokenizer facts affect the design:

• A visitor method cannot read the tokenizer offset. Return an error kind from the visitor and let the driver attach the offset.
• Escaped and unescaped strings use different callbacks. Keep escaped text borrowed. If a known member name must be compared, use the tokenizer's allocation-free unescaped comparison operation when available.
• Numbers arrive as validated JSON number text. Convert them only when the schema requires a value.
• The member separator is consumed by the tokenizer. A key callback is followed by the callback for its value.
• Confirm how complete input, trailing input, and sticky tokenizer errors are reported. Do not add a second completion model without a reason.

Choose the output shape from the interface

Streaming sequence. Let the consumer handle an item when the item is complete. This keeps peak memory near one item and avoids an intermediate document. A visitor or callback is often the best fit. An iterator adapter is useful when the caller requires the standard Iterator interface, but it should not add per-token work without a benefit.

Single struct. Keep only the fields needed to construct the value. Check required fields at the object boundary or another unambiguous completion point. Borrow fields from the input when their lifetime permits.

Enum or several document shapes. Use the earliest safe discriminator. If member order is not fixed, retain only the data needed to decide the variant. Do not build a full intermediate tree to postpone a decision.

Callback bridge. A generic bridge can borrow the caller's visitor while the decoder remains a concrete type. This can keep the public API simple and avoid storing a type parameter in decoder state. It is a useful option, not a rule. If profiling shows that a borrowed decoder adds repeated state indirection, test moving small plain decoder state by value across one drive call. One measurement in this repository found about a 7 percent difference on a small document. Treat that result as a clue, not a universal threshold. Keep the consumer borrow if it is touched less often than the decoder state.

Build the smallest safe machine

Use explicit states and a loop when the input can control nesting. Recursive descent turns input depth into call-stack depth and can exhaust the stack. Track only the context needed by the schema:

• a depth counter when only depth matters;
• compact fixed context when the container kind matters;
• a fixed capacity derived from a schema limit when a bound exists.

Unknown values may be skipped by counting nested containers in the decoder. Apply depth and size limits while skipping too. An unknown member must not provide an unbounded path around an input limit.

Emit a value as soon as it is complete. Keep a record only until its fields can be validated. Do not parse, unescape, copy, or store data that the result does not use. Use fixed-size inline storage for data with a real maximum. Reject values over that maximum; do not truncate or grow storage.

Name states after input positions, such as Parameters, AddressOctets, or Skip. A reviewer must be able to relate a state to a position in the input.

Correctness and cost rules

These are defaults because they preserve the tokenizer's properties. Change one only when the task changes the contract and the trade-off is explicit.

1. Do not allocate in non-test code. Avoid Vec, String, Box, collect, and to_owned. Borrow &str and other input data to the end of the call. Heap storage hides bounds and adds work that the tokenizer avoids.
2. Do not add broad decoding machinery by default. Avoid serde, runtime schemas, trait objects, and generic plumbing that is not required by the interface. They can add dispatch, compile time, binary size, or hidden allocation. Plain Rust keeps the hot path and its invariants visible.
3. Do not recurse. Input depth is external input, not a safe call-stack bound.
4. Handle input errors explicitly and do not use unsafe. Do not use unwrap, expect, panic!, todo!, unreachable!, unchecked indexing, or arithmetic that can overflow for input-derived conditions. A panic is appropriate only when reaching it would prove that an internal invariant is false, the invariant is known to hold in practice, and Rust cannot enforce it. Document that invariant and keep input validation before it. Use checked conversions, checked arithmetic, and bounds-aware access elsewhere.
5. Reject invalid data. Reject wrong types, values outside the target range, missing required members, and malformed state transitions. Apply a default only when the schema declares the member optional and documents the default. Skip unknown members only when the protocol permits it.
6. Make errors locatable. Preserve an error kind and pair it with the tokenizer byte offset at the driver boundary.

Document decisions in the generated module

Generated code needs its context in the file. Put this information in module documentation:

• what the decoder reads and where it comes from;
• the published schema;
• representative real sample documents;
• the public interface and why it has that shape;
• bounds, defaults, unknown-member policy, and caller responsibilities;
• the tests that verify the contract;
• an Editing this file section.

Start generated modules with:

//! GENERATED with the `kimojio-json-decoder` skill - prefer regenerating or
//! updating with that skill over hand-editing.

Document every public item. Comment only state transitions or invariants that are not clear from the code.

Separate the public surface from the machine

Public types, derives, accessors, error text, and documentation are ordinary source. A maintainer may edit them directly.

Mark the derived state machine, scratch values, visitor implementation, and entry-point or iterator bodies. Use markers like these:

// ---------------------------------------------------------------------------
// STATE MACHINE - maintained by the `kimojio-json-decoder` skill.
//
// Everything above this line is ordinary source.
//
// This region is derived to keep the path allocation-free, non-recursive, and
// explicit. A local change can break a transition, a bound, or an error path.
// Prefer describing the required behavior and letting the skill derive it.
// If this region is edited by hand, record why and re-check those properties.
// ---------------------------------------------------------------------------

// ------------------------- end of the state machine ------------------------

In Editing this file, say what is safe to edit by hand, what should be regenerated or updated by the skill, and which invariants must be checked after a machine edit.

Work modes

From scratch. Use the schema and interface as the source of truth. Design the smallest machine that meets them. Include the required module documentation and ownership markers.

Finish a sketch. Preserve the supplied public types and signatures unless the task says they are wrong. Complete the machine around that contract. Do not treat todo!() as permission to replace the interface.

Update. Read the module documentation and the machine markers first. Make the smallest change that satisfies the new contract. Do not regenerate a whole file for a local fix.

Assume that an existing file has hand edits. Preserve changes outside the machine. Read changes inside the machine before replacing them; an unusual branch or a fix comment records behavior that may be required. Carry that behavior forward or explain why it no longer applies. Summarize important design changes and any intentionally removed behavior.

Tests and validation

Keep hand-written acceptance tests independent. Do not edit or delete them to make the implementation pass. Tests inside the generated module may be added for local state-machine checks, but they do not replace the external contract.

Use the repository's existing validation commands. For a decoder package, the usual checks are:

cargo fmt -p <package>
cargo clippy -p <package> --all-targets --all-features -- -D warnings
cargo test -p <package>

After validation, inspect the machine against the allocation, depth, bounds, error-offset, and panic rules. These properties are the reason for the design; the exact state names and bridge layout are implementation choices.