run-benchmark

name	run-benchmark
description	Add a model to the GLAM extraction benchmark and run it, then refresh the leaderboard. Handles three model shapes — schema-native extractor Spaces (NuExtract), general VLMs via the HF Inference Providers router, and any OpenAI-compatible endpoint (including a model you served on Jobs). Use when adding or scoring a new model on this benchmark.

Run a model on the GLAM extraction benchmark

Each item = a card/form image + a canonical JSON Schema → JSON, scored by the model-agnostic typed-KIE scorer (glam_bench/scorer.py). Each model's solver converts the JSON Schema to its own dialect — so adding a model is mostly picking the right adapter.

1. Add the model to the registry

Edit glam_bench/models.py → MODELS, choosing a kind:

• router_vlm — any VLM on HF Inference Providers. Discover candidates:

  curl "https://huggingface.co/api/models?inference_provider=all&pipeline_tag=image-text-to-text"
  

  Pull cost/latency for the table from https://router.huggingface.co/v1/models/<id> (pricing.input).
• nuextract_space / specialist_space — a Gradio extractor Space.
• openai — any OpenAI-compatible /v1 endpoint: local vLLM, OpenAI, OpenRouter, or a model you served on Jobs (use the serve-on-jobs skill first). Set base_url, api_key_env, and guided: True to use vLLM schema-constrained decoding.

A registry entry: {"label": ..., "kind": ..., "id": ..., "params": ..., "cost": ...} (+ adapter fields).

2. Run + leaderboard

export HF_TOKEN=...                      # private dataset + Inference Providers / jobs endpoint
uv run glam_bench/harness.py --models <label1>,<label2>   # writes results/<label>.json (predictions + scores)
uv run glam_bench/leaderboard.py                           # content-F1 + per-field-type breakdown table

Notes

• Schema is JSON Schema (canonical). schema.py converts it to the NuExtract template or to guided_json per model. Verbatim fields are tagged {"x-match":"exact"} (the scorer reads it).
• Scores are silver/illustrative until human gold; NuExtract is advantaged because the silver gold was seeded from it. Read the breakdown (exact vs free-text, false-populate) not just the headline.
• Some models tagged multimodal fail to serve images via their provider — the harness records the failure (valid-JSON 0%) rather than hiding it. That's signal, not a bug.
• For a not-on-providers model (e.g. datalab-to/lift), serve it on Jobs first → see serve-on-jobs.