inference-layer

name

inference-layer

description

The unified model-inference layer under `services/inference/`. Read this before touching `run_inference`, any backend file (`gemini_api.py`, `gemini_cli.py`, `gemini_agy.py`, `codex.py`, `claude_sdk.py`), capability gating in `base.py`, the schema validate-and-repair loop in `schema_enforce.py`, or the agent frame tools under `services/inference/tools/` (`get_frames*.py`). Also read it when adding or renaming a `Backend`, or changing how audio/images/schema/prompts reach a model.

Unified inference layer (services/inference/)

One entry point for every model call in the repo:

run_inference(*, backend, prompt, system_prompt=None, cwd=None,
              images=None, audio=None, schema=None, model=None,
              reasoning_effort="high", output_last_message_path=None,
              timeout=None, web_search=False) -> InferenceResult

Five backends (Backend StrEnum in base.py):

Backend	Auth	Audio?	Schema handling	Cost
gemini-api	API key	✅	native response_json_schema	metered (the only paid backend)
gemini-cli	subscription	✅	prompt-appended + repair loop	free
gemini-agy	subscription	❌	prompt-appended + repair loop	free
codex	subscription	❌	prompt-appended + repair loop	free
claude	subscription	❌	prompt-appended + repair loop	free

Design rules baked into this layer — preserve them:

• One call, parameterized — not modes. schema=None → return the model's raw final text (agentic file-writing callers pass cwd and inspect files after). schema=<Model> → JSON guaranteed-parseable for that model.
• Capability gating lives in base.py (backend_supports_audio, is_agent_backend, is_gemini_backend). Passing audio to a non-audio backend raises UnsupportedMediaError. Callers must gate audio on the backend's capability, not just on whether an audio asset exists. is_agent_backend is "everything except gemini-api".
• web_search=True enables the backend's built-in web tools (agent backends only; gemini-api raises). Explicit enablement is required for codex (-c tools.web_search=true — off by default in codex exec; --yolo does not add the tool) and gemini-cli (policy allow rules for google_web_search/web_fetch); claude and gemini-agy already expose them under their permission bypass, so their runners treat the flag as a no-op. Every agent-instruction stage (pre-pass, chunk, refine, glossary check, date research) passes web_search=is_agent_backend(backend) (date research hardcodes True since its backends are always agentic).
• Schema enforcement is shared, not per-backend (schema_enforce.py): for the prompt-based backends, run_inference appends the schema instruction once and runs the validate-and-repair loop centrally. Each backend's only job is prompt → text. The retry cap is the hardcoded MAX_SCHEMA_RETRIES constant there (not a setting).
• Reasoning effort is repo-normalized: callers pass reasoning_effort as low/medium/high/extra. Backend wrappers map that to their real values: Codex and Claude use xhigh for repo-level extra; Gemini API and gemini-agy have no extra-high value and clamp extra to HIGH / High; gemini-cli exposes no separate effort flag, so only the model id is passed to that CLI.

Backend runtime gotchas:

• gemini_cli.py generates a temporary TOML policy file at runtime that whitelists only the frame-tool wrapper command prefixes, and uses include-dirs instead of --yolo (yolo's sandboxing breaks project-local frame reads). AGENT_GEMINI_GCP_PROJECT is injected as a temporary GOOGLE_CLOUD_PROJECT; API-key env vars are scrubbed from the subprocess so it never silently switches to API-key billing.
• CLI-shim subprocesses (gemini_cli.py, codex.py) must run via base.run_cli, which tree-kills on timeout (taskkill /T /F on Windows, killpg on POSIX) — plain subprocess.run(timeout=) kills only the cmd.exe shim and then blocks forever draining pipes the hung node grandchild still holds; do not revert to it.
• gemini_agy.py (Antigravity CLI) must run under a pty (pywinpty on Windows, stdlib pty on POSIX) — agy -p drops stdout on a non-TTY — and stages the prompt (and images) into a temporary workspace file referenced by @<file> tokens; the workspace is cleaned up after the call.

Agent frame tools (services/inference/tools/)

get_frames.py is a CLI agent backends run mid-session to extract up to 20 frames at specific --times for a moment they need to see. Stage-specific wrapper scripts (get_frames_for_pre_pass.py, ..._for_chunk.py, ..._for_refine.py, ..._for_glossary_check.py) pre-fill the stage and output directory; the agent should only pass --project-dir and replace the --times value. Extra frames land next to the stage artifacts: .pre_pass/media/extra_frames/, .chunks/media/extra_frames/, .refine/extra_frames/, .glossary_check/extra_frames/ — treat files there as the audit signal for whether the tool was actually used.

build_*_frame_tool_instruction renders frame usage; build_pre_pass_agent_instruction adds bounded agent-only web-search guidance. These blocks are appended to stage prompts only when is_agent_backend(backend) — gemini-api can't run local tools and never sees them.

Where to make a change

• New backend → add to Backend + capability sets in base.py, a run_* backend file, dispatch in run_inference (__init__.py).
• Test patterns for mocking backends live in tests/test_inference.py.