Evaluate LLM model quality using BERTScore, ROUGE, Perplexity, and Text Readability metrics on OCI AI Quick Actions (AQUA). Covers dataset preparation, evaluation job creation, and report interpretation. Triggered when user wants to evaluate or benchmark a model.
Instrucciones de origen · Vista previa de solo lectura
name
aqua-evaluation
description
Evaluate LLM model quality using BERTScore, ROUGE, Perplexity, and Text Readability metrics on OCI AI Quick Actions (AQUA). Covers dataset preparation, evaluation job creation, and report interpretation. Triggered when user wants to evaluate or benchmark a model.
user-invocable
true
disable-model-invocation
false
AQUA Model Evaluation
Use this skill when the user wants to evaluate LLM models on OCI Data Science using AI Quick Actions.
General text quality, aligns well with human judgement
ROUGE
N-gram overlap between generated and reference text
Summarization tasks
Perplexity
How well the model predicts the text
Language modeling quality
Text Readability
Reading level / complexity of generated text
Content accessibility
Dataset Format
JSONL format with required prompt and completion keys, optional category:
{"prompt": "Summarize this dialog:\nAmanda: I baked cookies...", "completion": "Amanda baked cookies and will bring some for Jerry tomorrow."}
{"prompt": "Translate to French: Hello world", "completion": "Bonjour le monde", "category": "translation"}
{"prompt": "What is 2+2?", "completion": "4", "category": "math"}
The category field dimensions evaluation metrics in the report (e.g., see accuracy per category).
When omitted, defaults to "_" (unknown).
Official sample datasets (10 prompts each, math + logic categories):
File
Use Case
examples/evaluation-sample-no-sys-message.jsonl
Llama-style prompts without system message
examples/evaluation-sample-with-sys-message.jsonl
Llama-style prompts with <<SYS>> system message
Python SDK Usage
Import
from ads.aqua.evaluation import AquaEvaluationApp
eval_app = AquaEvaluationApp()
