aoai-migration-evaluation

# Azure OpenAI Migration Evaluation Skill ## Purpose Run standardized evaluations to compare a current Azure OpenAI model against a candidate replacement and produce a go/no-go recommendation. Detect regressions before deploying a new model in production. Adopt a **continuous evaluation** approach to reduce migration costs over time. ## When to Use - Validating quality when migrating between Azure OpenAI models - Running A/B model comparisons (source vs target) - Setting up evaluation pipelines for model upgrades - Building golden datasets for regression testing - Running local or cloud-based evaluations - Establishing continuous evaluation to keep migration costs low across model generations ## Evaluation Architecture ``` ┌──────────────┐ ┌──────────────┐ │ Golden Dataset│───►│ Model A │──► eval_results_A │ (test cases)│ │ (current) │ │ │ └──────────────┘ │ │ ┌──────────────┐ │ │───►│ Model B │──► eval_results_B │ │ │ (candidate) │ └──────────────┘ └──────────────┘ │ Compare metrics Flag regressions ``` ## Pre-Built Evaluation Scenarios This repo provides four ready-to-run scenarios under `src/evaluate/scenarios/`: | Scenario | Module | Metrics | Test Cases | |---|---|---|---| | **RAG** | `src/evaluate/scenarios/rag.py` | Groundedness, Relevance, Coherence | 8 examples (policies, technical docs, legal, financial) | | **Tool Calling** | `src/evaluate/scenarios/tool_calling.py` | Tool Accuracy, Parameter Accuracy, Relevance | 8 examples (weather, calendar, email, search, stock) | | **Translation** | `src/evaluate/scenarios/translation.py` | Fluency, Coherence, Relevance | 10 examples (FR/EN/DE, business/tech/legal/medical) | | **Classification** | `src/evaluate/scenarios/classification.py` | Accuracy, Consistency, Relevance | 16 examples (sentiment, tickets, intent, priority) | ### Ready-to-Use Golden Datasets The `data/` directory contains **54 pre-built test cases** across 7 scenarios: | File | Cases | Scenario | |------|-------|----------| | `data/golden_rag.jsonl` | 10 | RAG / grounded Q&A | | `data/golden_classification.jsonl` | 10 | Intent & sentiment classification | | `data/golden_tool_calling.jsonl` | 8 | Function calling & tool selection | | `data/golden_translation.jsonl` | 6 | EN→IT/DE/ES translation | | `data/golden_summarization.jsonl` | 6 | Meeting notes, emails, incidents | | `data/golden_agent.jsonl` | 8 | Multi-step agent reasoning | | `data/golden_multiturn.jsonl` | 6 | Multi-turn conversation context | Use these as-is for quick validation, or as templates for your own domain-specific datasets. ### Quick Start — Run a Pre-Built Scenario ```python from src.evaluate.scenarios import create_rag_evaluator evaluator = create_rag_evaluator( source_model="gpt-4o", target_model="gpt-4.1", ) report = evaluator.run() report.print_report() ``` Other scenario factories: ```python from src.evaluate.scenarios import ( create_rag_evaluator, create_tool_calling_evaluator, create_translation_evaluator, create_classification_evaluator, ) ``` --- ## Two SDK Approaches — Critical Differences There are **two fundamentally different evaluation SDKs** offered by Microsoft. They differ in API surface, data mapping syntax, execution model, and SDK packages. Understanding these differences is critical before writing any evaluation code. ### Comparison Table: v1 (Local SDK) vs v2 (Cloud OpenAI Evals API) | Aspect | **v1 — Local SDK (`azure-ai-evaluation`)** | **v2 — Cloud Evals API (`azure-ai-projects`)** | |---|---|---| | **Package** | `pip install azure-ai-evaluation` | `pip install "azure-ai-projects>=2.0.0" azure-identity openai` | | **Latest version** | `azure-ai-evaluation>=1.15.0` (Feb 2026) | `azure-ai-projects>=2.0.0` (GA) | | **Execution** | Runs **locally** on your machine (Python process) | Runs **in Azure cloud** (server-side, async) | | **Entry point** | `from azure.ai.evaluation import evaluate` | `client = project_client.get_openai_client()` then `client.evals.create()` / `client.evals.runs.create()` | | **Evaluator specification** | Python class instances: `CoherenceEvaluator(model_config=...)` | Dict-based `testing_criteria` with `"evaluator_name": "builtin.coherence"` | | **Data mapping syntax** | `"${data.query}"` and `"${outputs.response}"` | `"{{item.query}}"` and `"{{sample.output_text}}"` | | **Config structure** | `evaluator_config` dict with `column_mapping` per evaluator | `data_mapping` dict inside each `testing_criteria` entry | | **Data source** | Local JSONL/CSV file path string | Uploaded dataset (via `project_client.datasets.upload_file()`) or inline `file_content` | | **Result logging** | Optional: pass `azure_ai_project` param to log to Foundry | Automatic: results always stored in Foundry project | | **Eval/Run separation** | Single `evaluate()` call does everything | Two-step: create eval definition → create run(s) against it | | **Agent evaluation** | Supports agent inputs via conversation format | Native agent targets (`azure_ai_agent`, `azure_ai_responses`) | | **CI/CD integration** | Run in any Python CI job | Cloud-native; poll for async results | | **Continuous evaluation** | Manual scheduling via cron/CI triggers | Native: `evaluation_rules` + `schedules` on `AIProjectClient` | | **Custom evaluators** | Any Python callable | Register via ML Client, or use prompt-based `azure_ai_evaluator` type | | **Grader types** | N/A (evaluators are Python classes) | `string_check`, `model_grader`, `azure_ai_evaluator`, `text_similarity` | | **Portal support** | Foundry classic portal | Both Foundry classic and Foundry (new) portals | ### Key Syntax Differences — Side by Side **Data mapping:** ``` v1 (local): "${data.query}" "${data.response}" "${outputs.context}" v2 (cloud): "{{item.query}}" "{{item.response}}" "{{sample.output_text}}" ``` **Evaluator reference:** ``` v1 (local): CoherenceEvaluator(model_config=model_config) # Python class instance v2 (cloud): {"evaluator_name": "builtin.coherence", ...} # String identifier ``` **Submission pattern:** ``` v1 (local): result = evaluate(data="data.jsonl", evaluators={...}) # Single call v2 (cloud): eval_obj = client.evals.create(...) # Step 1: define eval_run = client.evals.runs.create(eval_id=eval_obj.id, ...) # Step 2: run ``` --- ## When to Use Each Approach | Scenario | Recommended Approach | Why | |---|---|---| | **Quick local prototyping** | v1 (local SDK) | No cloud setup needed, fast iteration | | **CI/CD pre-deployment gate** | v2 (cloud) OR v1 with `azure_ai_project` | Cloud scales better; v1 can also log to Foundry | | **Large dataset evaluation (500+ rows)** | v2 (cloud) | No local compute limits; async execution | | **Continuous post-deployment monitoring** | v2 (cloud) | Native `evaluation_rules` and scheduling | | **A/B model comparison during migration** | v1 (local) or v2 (cloud) | v1 for quick iteration; v2 for production-grade | | **Agent evaluation** | v2 (cloud) | Native `azure_ai_agent` target support | | **Red teaming** | v2 (cloud) | Native `azure_ai_red_team` scenario | --- ## Approach 1: Built-in LLM-as-Judge (Quick, No Extra Dependencies) Uses `MigrationEvaluator` from `src/evaluate/core.py`. Calls both models, scores outputs with an LLM judge, and generates a comparison report. ```python from src.evaluate.core import MigrationEvaluator, TestCase evaluator = MigrationEvaluator( source_model="gpt-4o", target_model="gpt-4.1", test_cases=[ TestCase( prompt="What is Azure OpenAI?", system_prompt="You are a helpful assistant.", expected_output="Azure OpenAI is...", ), ], metrics=["coherence", "fluency", "relevance"], ) report = evaluator.run() report.print_report() report.save("migration_report.json") ``` Or pass a file path directly — loads JSONL automatically: ```python # File path variant — no need to construct TestCase objects evaluator = MigrationEvaluator( source_model="gpt-4o", target_model="gpt-5.1", # or "gpt-5.4-mini" for tier-down strategy test_cases="data/golden_rag.jsonl", # file path supported metrics=["coherence", "fluency", "relevance", "groundedness"], ) ``` --- ## Approach 2: Local SDK Evaluation — `azure-ai-evaluation` (v1) Uses Microsoft's built-in evaluator classes that run **locally in your Python process**. The `evaluate()` function accepts a JSONL file, instantiated evaluator objects, and optional column mappings using `${data.field}` syntax. ### Installation ```bash pip install azure-ai-evaluation # For cloud logging support: pip install azure-ai-evaluation[remote] ``` ### Built-in Evaluators Available (v1) | Category | Evaluators | |---|---| | **Quality (AI-assisted)** | `CoherenceEvaluator`, `FluencyEvaluator`, `RelevanceEvaluator`, `SimilarityEvaluator`, `GroundednessEvaluator`, `GroundednessProEvaluator`, `RetrievalEvaluator`, `ResponseCompletenessEvaluator` | | **Quality (NLP)** | `F1ScoreEvaluator`, `RougeScoreEvaluator`, `GleuScoreEvaluator`, `BleuScoreEvaluator`, `MeteorScoreEvaluator` | | **Safety** | `ViolenceEvaluator`, `SexualEvaluator`, `SelfHarmEvaluator`, `HateUnfairnessEvaluator`, `IndirectAttackEvaluator`, `ProtectedMaterialEvaluator`, `CodeVulnerabilityEvaluator` | | **Agent** | `IntentResolutionEvaluator`, `ToolCallAccuracyEvaluator`, `TaskAdherenceEvaluator` | | **Composite** | `QAEvaluator`, `ContentSafetyEvaluator` | ### Usage Pattern ```python import os from azure.ai.evaluation import ( evaluate, CoherenceEvaluator, FluencyEvaluator, RelevanceEvaluator, GroundednessEvaluator, ) # model_config points to the judge model (not the model being evaluated) model_config = { "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"], "api_key": os.environ.get("AZURE_OPENAI_API_KEY"), "azure_deployment": os.environ["EVAL_MODEL_DEPLOYMENT"], } result = evaluate( data="golden_dataset.jsonl", # JSONL file path evaluators={ "coherence": CoherenceEvaluator(model_config=model_config), "fluency": FluencyEvaluator(model_config=model_config), "relevance": RelevanceEvaluator(model_config=model_config), "groundedness": GroundednessEvaluator(model_config=model_config), }, evaluator_config={ "default": { "column_mapping": { "query": "${data.query}", # <- v1 syntax: ${data.field} "response": "${data.response}", "context": "${data.context}", } } }, # Optional: log results to Foundry portal azure_ai_project=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"), ) print(result["metrics"]) # Aggregate scores print(result["rows"]) # Per-row results ``` ### Logging v1 Results to Foundry Pass `azure_ai_project` to `evaluate()` to upload results: ```python result = evaluate( data="data.jsonl", evaluators={...}, azure_ai_project="https://<account>.services.ai.azure.com/api/projects/<project>", evaluation_name="migration-gpt4o-to-gpt41", tags={"migration": "gpt-4o-to-gpt-4.1", "environment": "staging"}, ) print(result.studio_url) # Link to Foundry portal results ``` ### A/B Comparison with Local SDK ```python from src.evaluate.local_eval import quick_evaluate, get_model_config, compare_local model_config = get_model_config() evaluator = create_rag_evaluator("gpt-4o", "gpt-4.1") source_items, target_items = evaluator.collect() result = compare_local( source_items, target_items,