foundry-evaluations

name	foundry-evaluations
description	Evaluate agent quality using Foundry OpenAI Evals API. Create evaluations, run them against models, and analyze results.
metadata	{"openclaw":{"requires":{"env":["FOUNDRY_PROJECT_ENDPOINT"]},"primaryEnv":"FOUNDRY_PROJECT_ENDPOINT"}}

Foundry Evaluations — OpenAI Evals API

You can evaluate agent and model quality using the Foundry OpenAI Evals API. Create evaluation definitions with testing criteria, run them against models, and analyze pass/fail results.

Endpoint

All requests: http://localhost:8443 with ?api-version=2025-11-15-preview. Auth is automatic.

Operations

List evaluations

curl -s 'http://localhost:8443/openai/evals?api-version=2025-11-15-preview'

Create an evaluation

curl -s -X POST 'http://localhost:8443/openai/evals?api-version=2025-11-15-preview' \
  -H 'Content-Type: application/json' \
  -d '{"name":"quality-check","data_source_config":{"type":"custom","item_schema":{"type":"object","properties":{"input":{"type":"string"},"expected":{"type":"string"}},"required":["input","expected"]}},"testing_criteria":[{"type":"string_check","name":"exact-match","input":"{{sample.output_text}}","reference":"{{item.expected}}","operation":"eq"}]}'

Run an evaluation

curl -s -X POST 'http://localhost:8443/openai/evals/eval_abc123/runs?api-version=2025-11-15-preview' \
  -H 'Content-Type: application/json' \
  -d