local-llm-architecture

name

local-llm-architecture

description

Use this skill whenever the user asks architectural questions about LLM systems, especially prompt design, agent/subagent responsibility boundaries, routing, retrieval planning, tool/capability design, reliability, latency, or how to divide work between an initializer/planner and specialized agents. This skill is particularly important for this repo because the target runtime uses a local/weaker LLM with finite context and chatbot latency constraints; invoke it even when the user does not explicitly mention local LLMs. When modifying an existing LLM pipeline, use this skill to audit existing contracts and minimize blast radius before editing.

Local LLM Architecture

Use this skill when reasoning about system-level design for this chatbot or any LLM pipeline where the deployed model is local, weaker, latency-constrained, or operating with limited context.

The goal is to avoid designs that look elegant with a frontier model but become brittle, slow, or expensive with the actual runtime model.

Core Assumptions

• The production model may be substantially weaker than the coding assistant.
• Context may be large but not effectively usable for schema discovery, multi-hop reasoning, or long instruction following.
• Chatbot responses need bounded latency. Multi-minute agent loops are unacceptable for normal user messages.
• The system should degrade gracefully when a query falls outside supported capabilities.
• The LLM should reason over semantic concepts, not raw database structure or implementation-specific schemas.
• NEVER send raw data to the LLM and expect it to infer the operational meaning. Local LLMs cannot reliably interpret numerical values without longer explanations that bloat prompts and still remain brittle. Always use a helper function to convert raw measurements into descriptive words before they enter the model context, for example messages_per_minute=6.2 -> noise_level="high" or distinct_speakers=5 -> speaker_diversity="many".

Design Posture

Prefer bounded, inspectable architecture:

• Start with the smallest semantic question the LLM must answer. Do not add a prompt input, output field, agent, retry, or repair path until a current requirement, observed failure, or approved near-term integration needs it.
• Preserve the existing pipeline contract before optimizing the new feature.
• Small set of explicit capabilities over many ad hoc tools.
• Typed or semi-typed intermediate representations over free-form prose when routing matters.
• Specialized agents own low-level parameter generation for their domain.
• Deterministic code owns validation, permissions, limits, schema mapping, and execution.
• Closed enum output values are for deterministic control only. If a generated value is consumed only as later LLM context, keep the key stable and validate the value as a short bounded semantic descriptor string instead of hard failing on unknown labels.
• The planner/initializer should decide what information is needed, not how to query every storage backend.

LLM Input Shaping

Before sending context to a local LLM, translate machine-facing data into semantic descriptors. The model should receive the interpretation it needs, not the raw telemetry it would have to calibrate from scratch.

Good payloads:

{
  "noise_level": "high",
  "speaker_diversity": "many",
  "reply_thread_fragmentation": "multiple_parallel_threads",
  "bot_turn_recency": "recent"
}

Risky payloads:

{
  "messages_per_minute": 6.2,
  "distinct_speakers": 5,
  "reply_target_count": 4,
  "messages_since_bot_last_spoke": 2
}

If numeric thresholds matter, keep them in deterministic helper functions where they are inspectable and testable. Pass only the resulting labels and short explanations to the LLM, such as "high_noise: many speakers and several parallel reply threads in the recent window".

Avoid designs where:

• A feature request causes changes to generic prompts, summarizers, finalizers, or persistence paths without an explicit architectural reason.
• A single initializer must understand every database shape and generate all low-level parameters.
• A weak local LLM is expected to infer schema or tool semantics from long prompts.
• A prompt receives routing, delivery, platform, permission, override, fallback, retry, cache, or feasibility inputs that deterministic code or an adapter should own.
• A retrieval agent can silently accept tasks outside its domain and produce plausible but wrong results.
• Retry loops are the default path for ordinary queries.
• More tools are added without clarifying semantic ownership.

Minimality Gate

Before proposing or editing any LLM prompt, schema, graph edge, or model call, write down the smallest current contract:

Semantic question:
Inputs required for that semantic question:
Output fields required by downstream code:
Deterministic owners:
Rejected complexity:
Evidence needed before adding complexity:

Reject proposed inputs that exist only to let the LLM decide operational behavior. If a field affects routing, delivery, platform behavior, permission, execution, override, fallback, retry, persistence, cache invalidation, or adapter feasibility, keep it out of the prompt and put the decision in the deterministic owner.

Universal path means one shared contract and one deterministic mapper; it does not mean one LLM stage receives every layer's operational context.

Responsibility Boundaries

Use this default separation:

User query
  -> lightweight planner / initializer: identify required facts and dependencies
  -> router: select the semantic capability or specialist
  -> specialist agent: generate low-level query parameters for one domain
  -> deterministic validator/executor: enforce schema, cost, scope, and safety
  -> answer synthesizer: explain results and uncertainty

The planner should produce a logical target such as:

{
  "intent_type": "user_list",
  "needs": "users whose display names end with 子",
  "constraints": [
    {"field": "display_name", "op": "ends_with", "value": "子"}
  ]
}

It should not need to know the physical storage layout, such as MongoDB field paths, aggregation syntax, index names, or dedup rules.

Existing-Pipeline Change Discipline

When modifying an existing LLM pipeline, start with a short system contract audit before editing. The goal is to avoid narrowing attention to the requested feature and accidentally changing the behavior of shared stages.

Before implementing, identify:

• Pipeline roles: what each existing stage is responsible for today.
• Allowed change surface: the smallest files/prompts/functions that must change.
• Forbidden change surface: generic prompts, finalizers, summarizers, persistence, cache invalidation, or repair paths that should not change unless explicitly required.
• Contract compatibility: whether the new capability follows the same initializer/router/specialist pattern as existing capabilities.
• Blast-radius risk: which existing tests, prompts, or live examples could be affected.
• Rejected complexity: which tempting fields, fallbacks, helper agents, retries, compatibility paths, or abstractions are intentionally excluded because the current semantic contract does not require them.

For this repo's RAG-style pipelines, keep this default invariant unless the user explicitly asks to redesign it:

initializer/planner -> semantic slot only
dispatcher/router   -> choose specialist by prefix/capability
specialist agent    -> extract low-level parameters for its own domain
deterministic code  -> validate schema, limits, permissions, and execute
summarizer/finalizer -> remain generic unless the requested fix is specifically there

Do not make the initializer build backend parameters simply because it can. If existing agents such as user-list or search agents extract their own arguments, a new agent should follow that local contract.

Do not add code-side repair, keyword routing, or semantic fallback in a shared orchestrator to compensate for a new agent. Put domain-specific extraction or refusal inside the specialist agent, and keep shared orchestration boring.

For local-LLM JSON output failures, keep recovery simple and proportional. Use the repo's existing JSON parser/repair helper when the JSON string itself is malformed. Once JSON parses, do not build retry loops, repair prompts, or extra LLM calls for missing fields unless the user explicitly asks for that recovery path. If a generated optional field is absent, skip that field; if a generated row lacks required fields, drop that row/data with a clear log and continue when the stage can safely degrade.

When the user says "put on the system engineer hat," treat that as a request to protect system contracts and minimize blast radius, not merely to make the new feature more structured.

Capability Design

When proposing tools or agents, define:

• Semantic ownership: what class of question this capability owns.
• Input contract: what structured arguments it accepts.
• Output contract: exact cardinality and shape, such as one user vs many users.
• Refusal conditions: tasks it should reject instead of attempting.
• Latency budget: normal number of LLM calls and database calls.
• Fallback behavior: whether to ask clarification, return partial results, or escalate to another capability.

Prefer capabilities like:

user_lookup        -> resolve one named user
user_list          -> enumerate users by metadata/predicate
message_search     -> find messages by keyword or semantic query
message_list       -> list messages by structured filters
memory_search      -> retrieve profile/persistent memories
profile_read       -> read one resolved user's profile
web_search/fetch   -> external information only

Do not overload one capability with incompatible cardinality unless the contract explicitly includes a mode such as lookup_one vs list.

Prompt Design Guidance

Prompts for local LLMs should be short, explicit, and contract-oriented.

System vs Human Message Boundary

Keep stable contract material in the system message. This includes role, standing policies, character-static context, capability semantics, allowed action names, output format, and the positive decision procedure.

Keep the human message for the current run only: user stimulus, upstream LLM outputs, retrieved evidence, tool results, and other dynamic facts. Do not put capability registry descriptions, handler affordances, stable examples, or repeated action semantics in the human payload. If the model needs the same information every run, it is system context.

For LLM-to-LLM handoff, prefer one precise semantic string over nested TypedDict-shaped payloads. Typed dictionaries, schema fields, IDs, and structured envelopes are for deterministic code unless the model is explicitly being asked to produce or consume that exact wire format. If the model-facing handoff cannot fit into one clear string without losing the essential task, stop and redesign the handoff instead of adding more nested fields.

Prompt Vocabulary Discipline

Never introduce a new idea, term, label, acronym, module name, stage name, plan phrase, or development-process concept into a persistent runtime prompt unless all of these are true:

• The term is already present in the model-facing input or is defined in the same prompt.
• The term directly helps the model answer the current semantic question.
• A plain domain phrase would be less clear.

If a term needs explanation, first try deleting it or replacing it with a plain semantic phrase. Runtime prompts should not contain implementation history, migration language, plan labels, review findings, compatibility-layer notes, checkpoint names, or comments about how the feature was developed. Those belong in design docs, tests, and code comments, not in prompts that the character model sees every turn.

Good prompts:

• Give a small roster of allowed capabilities.
• Use a positive decision procedure before any refusal rule.
• Provide examples only when they encode general reusable boundaries, not to force one known test case through a desired route.
• Ask for strict JSON only when downstream code validates it.
• Keep schema names semantic and stable.
• Use one language for ordinary instructions. Keep foreign terms only for keys, enum values, code, commands, URLs, model labels, or quoted source text.

Risky prompts:

• Long prose policies that require remembering exceptions far from the output format.
• Negative-constraint accretion: repeatedly adding "do not..." rules and counterexamples to chase individual failures.
• Test-shaped examples that mention the same nouns, route names, or expected answer as the live regression case being tuned.
• Mixed responsibilities, such as planning, routing, argument generation, and judging in one call.
• Free-form slot labels where a prefix can contradict the content.
• Instructions that require the model to infer database schema from examples.
• Dynamic human payloads that repeat stable capability definitions or registry metadata already known to the system prompt.
• Internal topology labels such as stage numbers, implementation module names, compatibility-layer names, or handler names unless those labels are part of the prompt's visible input contract and useful to the model's semantic task.
• Development artifacts in runtime prompts: plan names, migration phases, regression-test notes, temporary compatibility explanations, TODO language, or any wording whose only purpose is to explain how the code got here.

When editing prompt strings embedded in code:

• Define prompt constants with triple-single-quoted strings: '''...'''. Do not use triple-double-quoted strings, f-string prompt literals, f-string SystemMessage content, or .replace(...) prompt rendering.
• Render prompt templates only with .format(...) and named placeholders. .format(...) composition is allowed only for process-stable contract values; per-run facts still belong in the HumanMessage.
• Escape literal braces in .format(...) templates, or avoid inserting inline JSON examples there.
• Run a runtime prompt-render check, not only a syntax check. py_compile will not catch broken .format(...) placeholders.
• Keep examples as semantic boundary anchors. Add an example only when it fixes a reusable routing boundary or recurring confusion; do not add examples to make one fixture pass.
• For KV-cache-optimized local prompts, keep the SystemMessage static for the Python session. Static .format(...) composition is acceptable only for process-stable contract text. Put time, mood, user names, relationship state, retrieved evidence, tool results, and other per-run facts in the HumanMessage.
• Structured prompt stages must keep an explicit output contract. State # Output Format or the local-language equivalent, list JSON field names and types, and make the prompt body explain how each returned field is generated.
• # Input Format is optional after a prompt family has been approved for KV-cache input-format omission. When it is omitted, the remaining prompt body must explain each top-level human-payload field and decision-critical nested field where the prompt uses it. Do not treat the absence of # Input Format as a gap by itself.
• Include prompt-specific generation guidance before the output contract. The heading does not need to be exactly # Generation Procedure or # Thinking Steps; local headings such as # 审计步骤 or # 生成步骤 are fine when they tell the model how to inspect the input and choose output fields.
• Keep prompt vocabulary grounded in the prompt's visible inputs and role. Do not introduce foreign terms, acronyms, internal architecture names, storage jargon, or source labels unless they are present in the model-facing input or defined in the prompt body. If the handler payload contains rag_result.memory_evidence, say that exact field name or explain it semantically as "retrieved memory evidence"; do not say "RAG evidence" unless the prompt exposes and defines "RAG" as a concept the model should reason about. The same rule applies to terms such as "canonical", "planner", "router", or hidden node names: use them only when they are part of the model-facing contract, otherwise translate them into the evidence or decision source the model can actually see.
• Use one language for ordinary instructions. For free-form return fields that downstream stages read, state the expected output language; keep JSON keys, enum values, action names, code, commands, URLs, model labels, and quoted source text exact.
• Do not hard-code a character's concrete name in reusable prompt contracts. First decide whether the stage truly needs a name:
  • If the stage only needs role ownership, use role-neutral wording such as "the active character", "the character", or "the speaker". This is preferred for memory schemas, subjective appraisals, internal reasoning fields, RAG summaries, and any stage that might affect voice.
  • If the stage must distinguish the character from the user or resolve identity, inject the runtime value via a formatted variable such as {character_name}, following the local pattern used by cognition prompts. Never assume the character is always called by a name seen in logs or examples.
  • Avoid concrete character names in schema examples. Local LLMs can treat examples as voice anchors and drift into the wrong perspective, especially around subjective thinking, speech, and dialog-adjacent fields.

LLM Node Code Layout

When adding or editing LLM-backed nodes in this repo, keep each LLM stage readable as one local block:

prompt constant
LLM instance
LLM handler function

Do not group several prompt constants at the top of a file and several handlers far below when the stages are distinct. Do not collapse distinct LLM calls through a generic invocation helper that hides which prompt, model instance, payload, and parser belong to the stage.

Each handler should follow the established node style:

• Build SystemMessage(content=<PROMPT>) inside the handler.
• Build the HumanMessage from current-run dynamic content only. Use one semantic string for LLM-to-LLM handoff when possible; use json.dumps(..., ensure_ascii=False) only when the model-facing input contract genuinely needs structured dynamic fields.
• Call that stage's LLM with await <stage_llm>.ainvoke([system_prompt, human_message]).
• Parse structured JSON with parse_llm_json_output(response.content) unless the existing local file has a stronger approved parser contract.
• Run structural validation after parsing in ordinary Python.

If multiple stages share the same model config, it is still acceptable and preferred to declare separate stage-named LLM variables next to their prompts when that makes the prompt/handler ownership clearer.

Latency Strategy

Design for the common path:

1 LLM call to plan/route
0-N deterministic tool calls
optional 1 LLM call to summarize

Use repair loops only when:

• The query is high-confidence answerable.
• The failure is localized and cheap to repair.
• There is a hard iteration cap.

Prefer one-shot deterministic helpers for structured tasks such as user enumeration, exact filters, and profile reads.

Failure Handling

Make unsupported or ambiguous cases explicit.

Examples:

• If "user" could mean profiled users or observed chat participants, either choose the repo's default source or ask a clarification.
• If a message-search agent receives a user-metadata predicate, it should return an incompatible-intent error rather than searching message content.
• If a capability returns multiple candidates where a downstream step needs exactly one, stop and ask for disambiguation or use a documented tie-breaker.

Review Checklist

Before recommending an architecture or prompt change, check:

• Have I stated the smallest semantic question the LLM must answer?
• Have I written down the existing contract of the stages I am touching?
• Did I separate allowed change surface from forbidden change surface?
• Did I reject speculative fields, helper agents, retries, compatibility paths, and adapter/platform context that are not required by the current semantic contract?
• Would this still work with a weaker local LLM?
• Is the LLM being asked to infer hidden database structure?
• Is the LLM being asked to decide deterministic routing, delivery, permission, override, fallback, retry, persistence, or adapter feasibility?
• Is the responsibility boundary between planner, router, specialist, and executor clear?
• Does the new capability follow the same contract shape as neighboring capabilities?
• Can each agent reject out-of-domain work?
• Are cardinality expectations explicit?
• If a prompt mentions the character, did I verify that a concrete name is necessary? If not, use role-neutral wording; if yes, use runtime {character_name} formatting instead of a learned or hard-coded name.
• Is the normal latency path acceptable for a chatbot?
• Is there a deterministic validation layer before execution?
• Does the design improve one semantic capability rather than adding accidental complexity?
• Have I avoided changing generic summarizer/finalizer behavior unless that is the actual task?
• If I edited prompt templates, did I run prompt rendering checks in addition to syntax checks?