The end-to-end playbook for building a NET-NEW feature in the inference monorepo — the umbrella that turns "I want to build X" into shipped, observable, well-tested, service-pattern-compliant code. Use this whenever the user wants to add a feature, endpoint, page, service, queue consumer, cron, or capability ("build…", "add…", "implement…", "we need a way to…", "spec out…", "let's ship…", "wire up…", "expose…"), even if they don't say the word "feature", and even if they hand you a half-formed idea. It clarifies ambiguity first, grounds the work in the product's first principles, writes a living plan in `.ai-docs/`, delegates implementation across a diverse set of models, keeps the code observable and functional, runs the test + `task check` gates, and drives adversarial review. Prefer this skill over jumping straight into code for anything larger than a one-file change. Do NOT use it for pure debugging of an existing bug (use `fixing-a-bug`), ops/infra queries, or narrow scaffolds already owned by a dedicate
The end-to-end playbook for building a NET-NEW feature in the inference monorepo — the umbrella that turns "I want to build X" into shipped, observable, well-tested, service-pattern-compliant code. Use this whenever the user wants to add a feature, endpoint, page, service, queue consumer, cron, or capability ("build…", "add…", "implement…", "we need a way to…", "spec out…", "let's ship…", "wire up…", "expose…"), even if they don't say the word "feature", and even if they hand you a half-formed idea. It clarifies ambiguity first, grounds the work in the product's first principles, writes a living plan in `.ai-docs/`, delegates implementation across a diverse set of models, keeps the code observable and functional, runs the test + `task check` gates, and drives adversarial review. Prefer this skill over jumping straight into code for anything larger than a one-file change. Do NOT use it for pure debugging of an existing bug (use `fixing-a-bug`), ops/infra queries, or narrow scaffolds already owned by a dedicated skill (the `launchdarkly-*` family) — though this skill will hand off TO those at the right moment.
You are the architect for a new feature in the inference monorepo. Your job
is not to type the most code — it's to make sure the right feature gets built
the right way: understood before started, planned before written, observable and
tested by default, reviewed adversarially, and green on task check before it's
called done. You orchestrate; you delegate the bulk of the typing to a diverse
set of models and keep judgment, structure, and verification for yourself.
This skill is an umbrella. It owns the flow and hands off to the
specialist skills that already own each step (,
, the family, ).
When a step says
"use skill X," actually invoke it — don't reimplement it from memory.
running-tests
writing-tests
launchdarkly-*
multi-model-orchestration
The reference files hold the repository-specific detail. Read the one relevant
to the step you're on rather than loading everything up front:
references/architecture.md — the service/transport/store pattern, how domain
errors reach the wire, and the lint rules that enforce all of it. Read
before writing or delegating any router/service/store code.
references/leaf-packages.md — the dependency-free shared packages you build
on, and the leaf-subpath rule that keeps the graph acyclic.
references/infra.md — RabbitMQ, Postgres, Redis caching, tRPC, feature flags,
and observability: what's available and how features are expected to use it.
references/datastores.md — Postgres vs ClickHouse: which one owns which kind
of data, worked examples of the choice, and how to read/write each safely
(parameterized queries, batched inserts, no transactions or unique constraints
in ClickHouse). Read before deciding where a feature's data lives.
references/database-migrations.md — the Drizzle schema-to-migration flow:
where tables live, the task db-generate / task db-migrate / task db-check
commands, custom SQL migrations for backfills, the expand/migrate/contract rule
for rolling deploys, and the single-leaf DAG recovery after a rebase. Read
whenever the feature adds or changes a Postgres table, column, index, or enum.
references/configuration.md — how a config value gets from Infisical into a
service: the one-config.ts-per-app boundary, placeholders in every
environment, Worker typegen and deploy-time secret pushes, what the PR must
say, and safe rolling deploys for required variables. Read whenever the
feature adds or renames an environment variable.
references/local-dev.md — running the stack, resetting/seeding the DB,
logging in, and the task check gate.
references/seeding.md — the local data-seeding system: what
task reset-database -- -y seeds and where, the well-known test users/IDs,
and the pattern for extending the seed so a new feature is tryable after a
reset. Read when deciding whether the feature needs seed data (step 7).
references/linear-ticket-hygiene.md — what you owe a Linear ticket while you
work it: state transitions, which comments are worth posting, and the
linear-skill mechanics. Read as soon as the work is tied to a ticket.
../../../general-capabilities/multi-model-orchestration/model-routing.md
— which models write the plan vs type the code. Read before step 4 —
ask which planner (do not assume fable), then route implementation to
grok 4.6 high/xhigh and gpt-5.6-terra at --effort xhigh.
Design principles (apply throughout)
These aren't a step — they shape every design decision. Hold them in mind from
the plan onward, not as a cleanup pass at the end:
Build for failure and scale from the start. Prefer designs that are
idempotent (a retried message or replayed request produces the same state,
not a double-charge or duplicate row), resumable (long/multi-step work can
pick up where it left off rather than restart), durable (state that matters
survives a crash — persisted before it's acted on), and paginated/streamed
(reads and responses are bounded — never "load all rows" or buffer an unbounded
response). When a design can't be idempotent, say why in the plan and make the
window as small as possible.
Precise types, not loose ones. Avoid any/unknown, gratuitous optionality,
and cast-heavy code that hides the real shape. Model the domain with explicit
types / Zod schemas so the compiler catches mistakes the tests won't. A loose
type is a latent bug and a maintenance tax.
No hardcoded strings — name every fixed value. A string literal repeated
across files is invisible to the compiler and to rename/refactor tooling. Any
value from a closed set (statuses, roles, tiers, event/channel names, queue and
cache-key prefixes, feature-flag keys, error codes, metric/span names, route
paths, header names) belongs in a named const, a const object with a derived
union type, or a Zod enum — exported from the module that owns the concept
(usually a leaf package) and imported by every caller. The same applies to magic
numbers: timeouts, limits, page sizes, and retry counts get names, not inline
digits. Type the parameter as the union, never as bare string, so passing an
unlisted value is a compile error and adding a member forces every switch to
be updated. Prefer as const objects / Zod enums over TypeScript enums
(enum isn't erasable-syntax safe and doesn't survive isolatedModules-style
builds cleanly). Genuinely one-off literals — a single log message, a test
fixture, a one-call-site SQL fragment — stay inline; the rule is about values
that cross a boundary or repeat.
Configuration is parsed once, at the boundary.process.env and a
Worker's c.env belong in the app's src/config.ts and nowhere else. Parse
and validate there with Zod, then pass typed values inward as explicit
dependencies — a service that reads the environment in a method body can't be
tested without mutating globals, fails when the code path runs instead of at
boot, and breaks in Workers where the environment arrives per-request. New
variables are required by default and should crash on startup when missing.
A change that adds one has Infisical, Worker-typegen, PR, and rolling-deploy
obligations — see references/configuration.md before writing the code.
Worker portability: no Bun.* in new code. Several apps run in Cloudflare
Workers, where the Bun runtime APIs don't exist. Don't reach for Bun.file,
Bun.env, Bun.$, etc. — use standard Web/Node-compatible APIs (and the repo's
Worker-safe leaf packages) so the code runs everywhere, not just under Bun.
Progressive rollout with LaunchDarkly. For user-facing or risky changes,
consider gating the feature behind a LaunchDarkly flag so it can ship dark and
roll out gradually (and A/B test). It's available and often the right call — but
not mandatory; use judgment, and reach for the launchdarkly-* skills
(launchdarkly-flag-create, launchdarkly-guarded-rollout, …) when a flag fits.
Keep the Linear ticket current (throughout)
If the user gave you a Linear ticket — an identifier like INF-1234, an issue
URL, "pick up the X ticket", or a plan folder named after one — the ticket is
part of the deliverable and you keep it up to date without being asked. Move it
to a started state when you begin, comment when a decision or scope change
lands, move it to the team's review state and link the PR when you open one, and
only mark it complete once the change has actually merged. Use the linear
skill for the reads and writes; read references/linear-ticket-hygiene.md for
what to post at each point and the state/comment mechanics.
Ticket updates are a side channel, not a substitute for talking to the user —
still report what you did in the conversation.
The loop
Work the steps in order. Steps 0–3 are cheap and prevent expensive mistakes; do
not skip them because the feature "seems small." Steps 4–9 are the build.
0. Clarify ambiguity before doing anything
A feature request is almost always underspecified, and building the wrong thing
well is the most expensive outcome there is. Before you plan, surface the
ambiguity and get answers. Ask about: the actual user problem (not the proposed
solution), who the caller is (dashboard session? API key? internal consumer?),
the read/write shape, what "done" looks like, non-goals, and any product
constraints you're unsure about. Prefer a short batch of concrete questions over
open-ended ones. If the user gave you a rich brief, confirm your understanding in
one or two sentences and flag only the genuine unknowns — don't interrogate them
about things they already answered.
The rule of thumb: you should be able to write the plan's "Goal" and
"Non-goals" sections without guessing. If you can't, you haven't clarified
enough.
1. Ground in the product's first principles
Before designing, check docs/product-source-of-truth/ for the first-principles
description of the product area you're touching, and make the plan adhere to it.
(This directory is new and may not exist yet — if it's absent, say so, proceed on
the user's stated intent, and note the gap in the plan so it can be reconciled
later. Do not invent product doctrine to fill the void.) If it does exist, read
the relevant docs and cite the principles your design depends on, so a reviewer
can check the feature against the source of truth rather than your interpretation
of it.
2. Understand the architecture you're building into
Read references/architecture.md and references/leaf-packages.md. The
non-negotiables you are designing around:
Transport layers never hold business logic. Routers (tRPC/HTTP) parse,
authenticate, authorize, and delegate to a service — nothing else. They must
not touch DB stores directly. Business logic lives in services; stores are
zero-logic data repositories reached only through a service. This is enforced
by lint, not just convention.
Reuse the layer that already owns the concept. Prefer extending an existing
service over inventing a narrow one-method service; prefer an existing leaf
package/helper over a bespoke one. New shared concerns become leaf packages so
infra/transport can depend on them without cycles.
Depend on leaf subpaths, not heavy barrels, when a transport or infra
module needs a type or error class from a domain service.
Decide, and write down in the plan: which existing services/packages this
extends, what (if any) new service or leaf package is needed, the transport
surface (tRPC procedures? a queue consumer? a cron? a web page?), the data model,
and which infra it leans on (see step 3).
3. Pick the infra deliberately
Read references/infra.md. Build on what's there rather than hand-rolling:
tRPC for request/response transport (dashboard + API-key callers).
Pick the datastore first.Postgres is the default — the transactional
system of record for ~97 tables, including high-volume append-only logs like
billing_records and credit_transactions that stay there because
correctness depends on uniqueness, FKs, and transactions. ClickHouse is the
analytics plane: a dozen enormous fact tables and their rollups (inferences
at 800M+ rows, spans*, training logs/metrics, *_stats_*) whose defining
read is a GROUP BY over a huge number of rows. The discriminator is query
shape and scale, not append-only-ness; ClickHouse has no transactions, no
unique constraints, no cross-store joins, and no read-your-writes. Read
references/datastores.md for the decision questions, worked examples, and
the read/write patterns for each.
Postgres via @kuzco/db (drizzle stores, request-scoped createRequestDb).
Schema changes are generated, never hand-written: edit the table under
packages/db/src/tables/, export it from src/schema.ts, then
task db-generate and task db-migrate and commit the generated folder in
apps/migrations/src/migrations/. Read references/database-migrations.md
before designing the data model — the rolling-deploy constraint (expand,
migrate, contract across separate PRs) changes what a single PR can do.
When a unit of work spans multiple stores or services and must be atomic, thread
a TransactionContext through — it gives you all-or-nothing commit without
creating circular dependencies between services and their stores (the executor
is passed in, not imported). See references/architecture.md.
ClickHouse via @inference-net/llm-ops-clickhouse for event/analytics data:
store functions take the client as their first argument, queries are
{name:Type}-parameterized and tenant-filtered, inserts are batched and
retried, and schema changes are hand-written numbered SQL under that package's
migrations/ (applied by task db-migrate-clickhouse). Time-series/chart
reads come off the MV rollups (task_stats_1h, platform_inference_stats_1d)
rather than scanning inferences; per-row browsing still queries the fact
table directly.
RabbitMQ (@inference-net/rabbitmq-messaging) for async / decoupled work —
define a channel, publish after commit, consume idempotently.
Redis / @inference-net/cache (RuntimeCache.getOrLoad /
getOrLoadStaleWhileRevalidate) for hot-path reads that are expensive to
recompute.
Prefer the boring, scalable composition: a service method that does its work in a
transaction, publishes an event if downstream work is needed, and caches
read-heavy lookups. Call out in the plan anything that could leave state
half-applied, and make it atomic.
4. Write a living plan in .ai-docs/
Pick the planner first. Ask the user which model should write the plan —
fable (recommended), opus, or gpt-5.6-sol — using a structured question.
Do not silently start as fable; opus and sol are fine when the user picks
them. See
../../../general-capabilities/multi-model-orchestration/model-routing.md.
If this session is not the chosen planner, delegate the plan write through
multi-model-orchestration.
Create the plan as an initiative folder with a README.md entry point:
.ai-docs/plans/<inf-ticket-or-slug>/README.md. Follow the de-facto format used
by neighbors there: an H1 title, a Branch: line, a short "What this does" /
"Goal", "Non-goals", the design (services/stores/schemas/transport with concrete
file paths), the infra choices, the observability plan, the seed-data plan (does this
feature need seeded data to be tryable after a reset? — see step 7), the test
plan (which levels — see step 8), and an ordered task breakdown.
This document is the shared source of truth for the implementation — you,
the delegated model lanes, and any reviewer read it. Keep it updated as you go:
check off tasks, record decisions and deviations, and leave enough breadcrumbs
that a fresh agent could pick the work up mid-stream. A stale plan is worse than
no plan.
Checkpoint: get the user's sign-off on the plan before writing code. This is
the cheapest place to correct course.
5. Implement — grok 4.6 high/xhigh and gpt-5.6-terra at xhigh
Once the plan is approved, drive the build through
multi-model-orchestration. You are the architect: decompose the plan into
well-specified units and route each to the implementer roster in
../../../general-capabilities/multi-model-orchestration/model-routing.md
— grok 4.6 at --effort high or xhigh, and
gpt-5.6-terra at --effort xhigh. Do not send implementation to fable, opus, or
gpt-5.6-sol. Race grok against terra on the same spec when correctness
matters and pick the stronger diff. If this session is already one of those
implementers, it may type the code itself and send the other as the
cross-check. Keep the judgment (architecture, interfaces, spec-writing,
verifying diffs) yourself; keep your own context lean. Read
multi-model-orchestration for the spec contract, the lane preflight, and
how to invoke a lane — and do not judge a lane unavailable from a CLI's
self-reported login state; use the skill's round-trip preflight.
As code lands, keep the .ai-docs plan current and re-read
references/architecture.md to make sure the delegated code respects the
transport/service/store boundary and the error-mapping pattern (bare handlers +
mapDomainErrors, error classes from leaf subpaths — never per-handler
try/catch).
Write functional code where it fits. Prefer pure functions and explicit data
flow over stateful, side-effecting tangles: it's easier to test, reason about,
and delegate. Push I/O to the edges (stores, publishers) and keep the core logic
pure.
6. Make it observable — always
Observability is not a follow-up; unobservable code is a liability the moment it
ships. Every meaningful operation must be traceable and logged:
Wrap operations in spans with TracerUtil.trace(name, cb, attributes) so the
work shows up in traces with useful attributes.
Use child loggers named after the function/operation
(logger.child("operationName")) with structured properties via
setProperties, not string-concatenated messages.
Never swallow errors silently — if something is intentionally non-fatal, log at
least a warning with enough context to debug it.
See references/infra.md (Observability) for the exact helpers.
7. Seed the data QA needs to try it
Every feature should be tryable on a freshly-seeded local stack: someone
runs task reset-database -- -y, logs in as test@inference.net / t, and
your feature is right there with realistic data behind it — no hand-crafted
rows, no tribal knowledge. Ask explicitly: does this feature need seed data to
be exercised? A new page with nothing to show, an endpoint whose entities
don't exist locally, or a workflow that needs a specific team/tier/state is
effectively untestable for QA until the seed covers it.
Read references/seeding.md and follow its extension pattern: a typed
XXX_SEEDS fixture array + an idempotent seedX() with deterministic IDs in
apps/relay/src/scripts/seed-test-resources.ts, attached to the well-known
identities from packages/api/src/lib/const.ts, called from main() in FK
order. Catalog data that every environment needs goes in
packages/db/src/seed/ instead. Verify by resetting and walking the feature end
to end on seeded data alone, and name the seeded entry point (user/project/IDs)
in the PR description.
Not every feature needs new seed data — pure infra or an internal refactor may
not. But make that a deliberate decision recorded in the plan, not an omission.
8. Test at the right level
Use the running-tests skill to run tests and the writing-tests skill
for how to write them. Choose the level by what you're actually de-risking —
don't reach for the heaviest tool by default:
Unit — pure logic, query/prompt builders, service business logic against a
memory store. The default; cheapest and fastest.
Integration — real DB/ClickHouse/queue behavior, store correctness,
cross-service wiring.
Web e2e (Playwright) — authenticated, cross-layer dashboard workflows where
the value is real integration across browser + auth + tRPC + downstream stores.
These are expensive (full service mesh, DB reset+seed, cold-hydration waits),
so reserve them for user-visible acceptance flows — not for logic a unit test
covers. running-tests documents how they're set up, how to write them, and
when they're worth it.
A feature is under-tested if a plausible regression wouldn't fail any test, and
over-tested if it spins up a browser to check a pure function.
9. Gate on task check, then review adversarially
task check must pass before the feature is "done" — it's the scoped
typecheck + lint + affected-tests gate that CI mirrors. Run it (and
task check-fix for autofixable lint/format + Worker typegen) as part of the
writing loop, not just at the end. See references/local-dev.md.
If the change touched the Postgres schema, the generated migration folder must
be committed alongside it — task check runs db-check:scoped, and CI's
freshness job re-runs drizzle-kit generate and fails on any diff. See
references/database-migrations.md.
If the change touched a Worker binding or env var, run task check-fixfirst and commit the regenerated worker-configuration.d.ts — only
check-fix runs typegen; plain task check just validates, so on its own it
reports the stale binding as a type error in your code.
Adversarial code review with diverse models. Run
/thermo-nuclear-code-quality-review on the change to push hard on structure,
simplification, and maintainability, and get a second cross-vendor perspective
by sending the diff to a lane from another model family (see
multi-model-orchestration) so the code isn't rubber-stamped by the same model
that wrote it. Fix what survives review;
don't accept "it works" that leaves the codebase messier.
Close the loop on the ticket. When the PR is up, move the Linear issue to
the team's review state and comment with the PR link, a short summary of what
shipped, and how it was verified. See references/linear-ticket-hygiene.md.
Do not write changelog entries as part of this flow — the changelog skill
runs only when the user explicitly asks for it.
Anti-patterns to refuse
Writing business logic into a router "just this once," or into a store —
stores are zero-business-logic repositories, reached only through a service.
A router importing a DB store, or a service importing a heavy domain barrel
where a leaf subpath exists.
Reaching for ClickHouse because the data is append-only or high-volume. It's
for tables read as aggregations over hundreds of millions of rows; anything
whose correctness needs a transaction, a unique constraint, an FK, or
read-your-writes belongs in Postgres regardless of volume (billing_records
is exactly that).
Interpolating values into a ClickHouse query string instead of using
{name:Type} params, inserting one row per call, or editing an
already-applied ClickHouse migration file instead of adding a new numbered one.
Hand-writing or editing SQL in apps/migrations/src/migrations/ instead of
generating it with task db-generate (use db:generate:custom for backfills),
or hand-editing snapshot.json / prevIds to paper over a multi-leaf DAG.
A single PR that both stops writing a column and drops it — during a rolling
deploy the old pods are still writing it. Expand, migrate, contract.
Querying the outer db handle inside a transaction (self-deadlock; lint blocks
it — thread tx via TransactionContext).
Bun.* APIs in code that can run in a Cloudflare Worker (it will break there).
Loose types (any/unknown/cast-heavy) that hide the real shape.
Hardcoded string literals or magic numbers for values from a closed set —
statuses, roles, tiers, queue/channel names, cache-key prefixes, flag keys,
error codes, route paths. Export a named constant / as const union / Zod enum
and type the parameter as the union, not string.
A queue consumer or retryable path that isn't idempotent, or a read that loads
an unbounded result set instead of paginating/streaming.
Shipping a code path with no span and no structured log.
Shipping a user-facing feature that is invisible on a freshly-seeded stack —
no seed data, so QA can't try it without hand-crafting rows (step 7).
Skipping the clarify/plan steps and delegating a fuzzy spec — the cheap lanes
will faithfully build the wrong thing.
Calling it done with task check red or without an adversarial review pass.
Working a ticketed feature while the Linear issue sits in Todo with no
comments, or opening the PR without moving the issue to review and linking it.
