ml-evaluation

name	ml-evaluation
description	ML evaluation: lm-eval-harness benchmarks (MMLU, GSM8K, etc.) and Weights & Biases experiment tracking.
version	1.0.0
author	Hermes Agent
license	MIT
platforms	["linux","macos"]
metadata	{"hermes":{"tags":["ML","evaluation","benchmarks","experiment-tracking","W&B","lm-eval"]}}

ML Evaluation Suite

Unified skill for evaluating and tracking ML experiments.

---

Section A: lm-eval-harness (LLM Benchmarks)

Benchmark LLMs on standard tasks: MMLU, GSM8K, TruthfulQA, HellaSwag, ARC, Winogrande, etc.

Best for: Comparing model performance across standard benchmarks, evaluating fine-tuned models.

Key benchmarks:

• MMLU: Massive Multitask Language Understanding (57 subjects)
• GSM8K: Grade school math word problems
• TruthfulQA: Truthfulness in generation
• HellaSwag: Commonsense natural language inference
• ARC: AI2 Reasoning Challenge
• Winogrande: Adversarial pronoun disambiguation

Typical workflow:

pip install lm-eval
lm_eval --model hf --model_args pretrained=model_name --tasks mmlu,gsm8k --batch_size auto

---

Section B: Weights & Biases (Experiment Tracking)

Log ML experiments, sweeps, model registry, and dashboards.

Best for: Tracking experiments across runs, hyperparameter sweeps, model versioning, team collaboration.

Key features:

• Experiment logging (metrics, hyperparams, artifacts)
• Hyperparameter sweeps (grid, random, bayesian)
• Model registry
• Dashboard creation
• Team collaboration

Typical workflow:

import wandb
wandb.init(project="my-project", config={"lr": 0.01})
wandb.log({"loss": 0.5, "accuracy": 0.9})
wandb.finish()