Menu
Evaluation provider abstraction for dataset experiments (Langfuse today). Use when running an experiment, adding a new eval backend, or wiring evals into CI.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Router for any non-trivial echo-sdk task. Maps user intent to the right module skill and diagram. Invoke FIRST for any change to src/echo/ before reaching for a specific module skill.
Recipe for adding a new provider (LLM, transcriber, eval, prompt). Use whenever adding Cohere, Mistral, Deepgram, Phoenix, Promptlayer, etc. — anything that plugs into an existing get_*() factory.
End-to-end recipe for defining a Skill bundle and wiring it into a GenericAgent with the right activation mode and tool registry. Use when extending agent capability via swappable skills rather than baking tools in.
Building agents with BaseAgent / GenericAgent, including skill registration, activation modes (llm vs manual), and system-prompt composition. Use when editing src/echo/agents/ or building a new agent subclass.
Audio transcription provider abstraction (Gemini, EkaCare). Use when transcribing audio, adding a new transcriber, or shaping AudioInput.
Postgres support — asyncpg client, schema-aware binder, PgQueryTool surface. Use when querying postgres from a tool, modifying the binder, or adding a new DB engine.
| name | echo-sdk-evals |
| description | Evaluation provider abstraction for dataset experiments (Langfuse today). Use when running an experiment, adding a new eval backend, or wiring evals into CI. |
BaseEvalProvider in evals/base.py — abstract; declares run_experiment(name, dataset_name, run_func).LangfuseProvider in evals/langfuse_provider.py — concrete implementation.get_eval_provider() in evals/factory.py — singleton factory.run_func signature: async def run_func(item) -> result — receives a dataset item, returns the model output (and optionally metadata). Provider records the result against the experiment.langfuse is an extra; guard imports.run_experiment.[[echo-sdk-prompts]]) — label="production" drifts under your feet.→ [[echo-sdk-adding-a-provider]]. Implement BaseEvalProvider, register in factory.py.
run_func invocations — items may run concurrently; keep run_func pure.run_func — wrap in asyncio.to_thread or skip.examples/prompt_eval_usage.py[[echo-sdk-prompts]], [[python-optional-deps]]