| name | together-evaluations |
| description | LLM-as-a-judge evaluation framework on Together AI. Classify, score, and compare model outputs, select judge models, use external-provider judges or targets, poll results and download reports. Reach for it whenever the user wants to benchmark outputs, grade responses, compare A/B variants, or operationalize automated evaluations. |
Together AI Evaluations
Overview
Use Together AI evaluations when the user wants a managed LLM-as-a-judge workflow rather than an
ad hoc prompt loop.
Core evaluation types:
- Classify: assign outputs to labels
- Score: grade outputs on a numeric scale
- Compare: compare two candidate outputs with bias controls
This skill also covers external providers used as judges or targets when the workflow still runs
through Together AI's evaluation system.
When This Skill Wins
- Benchmark prompt variants, models, or product responses
- Grade quality, safety, policy compliance, or task success
- Run A/B comparisons between model outputs
- Build repeatable evaluation jobs with uploaded datasets
- Pull results programmatically after asynchronous execution
Hand Off To Another Skill
- Use
together-chat-completions for one-off inference or manual judge prompts
- Use
together-batch-inference for bulk offline generation rather than evaluation
- Use
together-fine-tuning when the user wants to improve the model instead of just measure it
- Use
together-dedicated-model-inference only if the evaluation target itself is a dedicated endpoint
Quick Routing
- Classify / Score / Compare job setup
