failproofai-sdk

name

failproofai-sdk

description

The way to make an AI agent report what it did to Failproof AI — planning what to record, writing the instrumentation, and proving the events land. Reach for it on vague phrasing too: "add observability to my agent", "why isn't my agent showing up?" Trigger when the user wants to: • plan an integration — which points in their agent loop to record, and what the platform must see before sessions, errors, and evals work at all; • write or fix instrumentation — add the `failproofai_sdk` Python SDK to an agent codebase, thread session/agent identity through it, emit tool, model, hook, or human events; • verify it — confirm events are being written, or debug an integration that looks correct and produces nothing. Served by the `failproofai_sdk` Python SDK, inside the user's own agent. NOT for reading telemetry that already landed or operating a deployment (that's `fp-cloud-cli`), or building the evaluator service that scores runs (that's `agenteye-evaluator`).

Failproof AI Python SDK

The SDK records what your agent did, from inside your agent. You call it at points you choose; it appends structured events to local .jsonl files. A separate collector ships those files to the platform.

your agent calls failproofai_sdk.event.*
  → SDK queues it in memory
  → flush thread writes <base_dir>/events/event-<timestamp>.jsonl
  → collector picks the file up and ships it
  → visible as sessions / events / errors / evals

The SDK's job ends at the file. That boundary is the most useful thing to know about it: everything up to the .jsonl is yours to get right and yours to verify, and it is verifiable on a laptop with no server, no API key, and no network.

The API is small — 15 event methods, all keyword-only. The hard parts are deciding where to call them and knowing which silences are bugs, because this SDK does not raise when you get it wrong. Sections 1-3 are the plan, 4 is the code, 5-6 are the proof.

1. Install it

pip install failproofai-sdk        # or: uv add failproofai-sdk

The distribution is failproofai-sdk and the import is failproofai_sdk. Public PyPI, no token, no dependencies.

One command to never run: pip install agenteye. That name belongs to a stranded release of an old CLI — a different product that shipped under it before moving to fp-cloud-cli. PyPI versions cannot be withdrawn, so the name still resolves to that build forever. You get the CLI, import failproofai_sdk raises ModuleNotFoundError, and on a codebase still using the pre-rename SDK (which published under agenteye too) pip treats it as an upgrade and removes the SDK.

Tell: if a coding agent proposes pip install agenteye to install the SDK, this skill never loaded. Stop and re-read it.

The CLI is a fine thing to want — it is what reads the telemetry back. Install it separately, never with pip into your agent's environment:

pipx install fp-cloud-cli        # the command is `fp`

Confirm what you actually have before writing a line of instrumentation:

python -c "import failproofai_sdk; print(failproofai_sdk.__version__)"

A version like 0.0.1b1 is the SDK. ModuleNotFoundError means it is not installed — check pip show agenteye, which returning anything means the wrong name was installed. references/install.md covers migrating an existing import agenteye integration.

2. Plan before you instrument

Instrumentation lands in code that already exists and already works. Read it first, then decide. Two questions settle most of the design, and only the user can answer the first:

What is one run of this agent? That is your session_id — one value for the whole run, generated by you at the point the run starts. A chat turn, a job, a request, a workflow execution. If the agent handles concurrent runs, this must be per-run, not per-process.

What are the distinguishable actors in a run? That is your agent_id — a stable label, not a unique id. "planner", "researcher", "main". It is how the platform tells sub-agents apart, so reuse the same string across runs.

Get these two named and agreed before writing code. They are the axes every surface groups by, and changing them later splits the history: old runs keep the old labels and the trends break.

The two events everything else hangs off

Most of the catalog is optional and incremental. These two are not:

Event	Without it
agent_start	The session does not exist. No row on Sessions, no timeline, no evaluation — while every other event you emit still lands fine and shows up in the event stream.
agent_end	The run never closes, and it is not handed to the evaluator at the normal time.

That first row is the single most common integration failure, and it is completely silent: a run emitting 500 tool calls and no agent_start produces a busy event stream and zero sessions. Sessions are defined as "something that emitted agent_start". So:

Emit agent_start at the top of the run and agent_end at every exit, and get those two working end-to-end before you instrument anything else. One event at each end proves the whole path — install, identity, base dir, collector — with almost no code to be wrong. Add tools, models, and hooks after that path is green.

Then map the rest onto the agent's shape

Walk the agent loop and pick the points that exist in this codebase. Skip what doesn't apply; there is no requirement to emit every type.

In the code	Emit	Buys you
every exit path of a run — success, exception, early return	agent_start / agent_end	the session itself
the tool dispatcher, both sides of the call	tool_use / tool_result	what ran, in what order, how long
the LLM client wrapper, both sides	model_request / model_response	model mix, token spend, stop reasons
your except blocks	error	the Errors surface
a policy/guard/middleware layer	hook_triggered / hook_completed	hook behaviour
an approval gate or human handoff	human_wait / human_input, human_pause, human_interrupt	where runs sit waiting on people
a run that suspends and resumes — waiting for a human, throttled, user-paused	agent_pause / agent_resume	a real "paused" state: the agent isn't ended, the resume isn't a new agent, and wait time is excluded from active work

If the codebase has one tool dispatcher and one LLM wrapper, you have two edit sites for the bulk of the value. If tool calls are scattered inline across the codebase, say so — a wrapper (§4) is worth more than 40 call sites.

Full field-by-field catalog: references/events.md.

3. The contract

Work with these; none of them raise, so none of them show up in testing.

• There IS an ambient session, and it is the ergonomic path. session(), agent() and tool_call() bind identity on contextvars, so session_id and agent_id are optional on all 15 event methods — omitted, they resolve from the enclosing scope. current() reads it; propagate(fn) carries it into a new thread, which contextvars do NOT do on their own.

  This section said the opposite until the scopes existed, and the reference integration shipped a contextvars wrapper as markdown for customers to paste into their own code. That is now in the package.

  Nothing bound and nothing passed raises TypeError naming the fix — never a silent emit, because ingest skips an event with no session and answers 200.

  Two more shapes raise, for the same reason:

  You pass	Raises	Why it cannot be allowed through
  A non-str id	TypeError	Ingest skips the event and still answers 200
  "" or " "	ValueError	Worse — ingest accepts it, and every event merges under one blank id

• configure() is optional, and every call restates all of it. It is keyword-only with exactly three settings:

  arg	default resolution
  base_dir	~/.failproofai/custom-agents (honours $FAILPROOFAI_HOME)
  environment	$AGENTEYE_ENVIRONMENT, else "dev"
  flush_interval	0.5 (seconds)

  No environment variable can move the spool out of the umbrella. $FAILPROOFAI_HOME relocates the umbrella itself, but custom-agents is appended unconditionally, so the spool is always inside it. base_dir is the only way to write anywhere else, and it is an explicit argument at the call site rather than something inherited from the environment.

4. Write it

Threading session_id and agent_id through every call site by hand is the thing that makes integrations ugly and abandoned. Don't. Bind identity once per run and let the call sites read it.

references/frameworks.md covers the four adapters. references/integration.md has the hand-written wrapper — one small module, correct under asyncio and threads, adaptable to any codebase — plus worked shapes for a tool dispatcher, an LLM client wrapper, and framework-specific callback layers. Read it before writing your own; the naive version (a module global, or a plain attribute) breaks the moment two runs overlap, and it breaks by mixing two runs' events together rather than by failing.

Match the codebase you're in. If it's async, the wrapper is async. If it already has a request context or a trace id, bind to that instead of inventing one.

5. Verify — watch the files

This is the whole point of the file boundary: you can prove the integration without a server. Run the agent and look.

Resolve the spool the way the SDK does, rather than guessing at a path:

python -c "import failproofai_sdk._resolver as r; print(r.get_base_dir() / 'events')"

That prints ~/.failproofai/custom-agents/events unless the application called configure(base_dir=...). $FAILPROOFAI_HOME moves the ~/.failproofai part and nothing else. $AGENTEYE_HOME does not affect it — that variable belongs to the older agenteye-collector, which reads it to decide what to WATCH.

ls -la ~/.failproofai/custom-agents/events/

You are looking for event-<UTC timestamp>-<pid>-<seq>.jsonl files — the pid and sequence number are what keep two processes flushing in the same millisecond from overwriting each other. Each line is one event. Read them with a JSON parser, not grep — the exact spacing is not a contract, and a grep for "type":"agent_start" returns nothing on a perfectly healthy integration:

cat ~/.failproofai/custom-agents/events/*.jsonl | python -m json.tool --json-lines | head -20

Then check, in this order — the first failure explains everything downstream:

1. Any files at all — or do they stop mid-run? Look at stderr for Exception in thread failproofai-sdk-flush. This is the first thing to check and the worst thing to miss: one non-JSON-serializable value killed the writer, and everything after it — including the at-exit flush — is gone (§3). The tell is that events stop for every type at once, and nothing raised. If instead there were never any files: did import failproofai_sdk succeed (§1)? Is the base dir writable? Did the process die hard (SIGKILL, docker stop, an OOM) before a flush?
2. Is agent_start there, once per run? No → you will see events on the platform and no sessions, and you will spend an afternoon on it (§2).
3. Sessions but no tool or model events? Your emit path is dropping them before the SDK ever sees them — nearly always because they're emitted from a thread the identity never reached. See references/integration.md → "Threads will drop your events". The SDK is silent here; only your own wrapper can warn.
4. Is environment what you expect? It is "dev" unless you set it (§3).
5. Is outcome on agent_end a word that counts? failed/error/timeout/ rejected — not "failure" (§3). Failed runs showing green is this, every time.
6. Run two overlapping runs. Confirm two session_ids with no events crossing between them. Do not check this with one run: a single run passes even when identity is a module global, and mixing only appears once two runs overlap — which is production, not your laptop (§4).
7. Do tool_use and tool_result share a tool_call_id? Unpaired means no duration. Also confirm your ids are unique process-wide — a collision pairs the wrong two events and reports a confident wrong duration (§3).

A test-mode loop that costs nothing:

export FAILPROOFAI_HOME=/tmp/failproofai-sdk-test
rm -rf /tmp/failproofai-sdk-test && python your_agent.py
cat /tmp/failproofai-sdk-test/custom-agents/events/*.jsonl | python -m json.tool --json-lines

FAILPROOFAI_HOME sends events somewhere disposable, so you can iterate on the integration without touching the real directory or shipping test runs to the platform. Note the custom-agents segment in the read path — the SDK appends it unconditionally. Note too that the SDK reads the variable late, per flush, so set it before you start the process, not halfway through.

AGENTEYE_HOME used to do this job and no longer does anything to the SDK; using it here would write to your REAL spool while you read an empty temp directory.

Do not verify by installing the CLI into your agent's environment. It will uninstall the SDK you just integrated (§1). Reading back what landed on the platform is the fp-cloud-cli skill's job, from a separate environment.

6. Production — the collector has to agree with you

The SDK writes files. It never talks to the network, so from its point of view a completely unshipped integration looks perfect.

In production, the collector must be running and reading the same directory the SDK is writing to. That is the whole contract, and both halves fail silently:

• Collector not running → files pile up in events/ forever. The SDK is fine.
• Collector reading a different base dir than the agent writes to — the collector's own AGENTEYE_HOME pointing somewhere else, a different user's ~, a container path that isn't mounted → files pile up in a directory nobody reads. The SDK is fine. (failproofaid watches both ~/.failproofai/custom-agents/events and ~/.agenteye/events, so it is the half of this pair least likely to be misconfigured.)

So when events are on disk but not on the platform, the SDK is not the suspect. Compare the two paths first: print the directory your agent is actually writing to (python -c "import failproofai_sdk._resolver as r; print(r.get_base_dir())" in the agent's own environment, with the agent's own env vars) and check the collector is running and pointed at the same one. A .jsonl count that only grows is the tell.

Confirming events arrived on the platform is deliberately not this skill's job — that is the fp-cloud-cli skill, from a separate environment (§1). Collector setup and deployment are your platform's own documentation.

If the files look right (§5) and the collector is running against the same directory, the integration is done.