0xSero/model-toolkit

Dernière activité source enregistrée: 10 septembre 2026 à 09:13

Catalogue SkillsMP mis à jour: 5 novembre 2026 à 19:57

skills collectés: 13

Étoiles GitHub: 4
Forks GitHub: 1

Installer avec un assistant IA

Copiez ce prompt dans l’assistant IA que vous utilisez.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/0xSero/model-toolkit.

README du dépôt

Fichier source du README : README.md

Lire le README source

Afficher le README complet

# Model Toolkit Practical tools for evaluating, observing, pruning, and quantizing language models. Each tool also includes a portable `SKILL.md` for coding agents. Pick a folder, install that tool's dependencies, and run it. There is no server or orchestration framework to deploy. This repository collects the workflows used in 0xSero's REAP and EXL3 experiments, including the GLM-5.3 fidelity campaign and Harbor coding evaluations. It contains reusable Python tools plus small launchers for the original upstream implementations. Model weights, benchmark questions, private session logs, and credentials are not bundled. ## Choose a tool | I want to… | Tool | What runs | |---|---|---| | Test an agent on terminal tasks | [Terminal-Bench](terminal-bench/) | Harbor; explicit **2.1** default, configurable dataset | | Test long software engineering tasks | [DeepSWE](deep-swe/) | Official DeepSWE task set through Pier | | Measure graduate-level science reasoning | [GPQA Diamond](gpqa-diamond/) | Included API runner, deterministic choice shuffle, retained responses | | Evaluate scientific code | [SciCode](scicode/) | Official SciCode Inspect evaluator | | Run reasoning/knowledge/ instruction benchmarks | [LM evaluation](lm-evaluation/) | EleutherAI task registry: MMLU, GSM8K, IFEval, and more | | Measure fidelity lost through pruning/quantization | [KL divergence](kl-divergence/) | Included full-logit capture and aligned forward-KL comparison | | Collect expert activation statistics | [REAP observation](reap-observation/) | Pinned Cerebras REAP observer; observation-only by default | | Select which experts to keep | [REAP pruning](reap-pruning/) | Global, domain-balanced, and max-over-domain keep plans | | Convert a supported model to EXL3 | [EXL3 quantization](exl3-quantization/) | Pinned ExLlamaV3 converter, recipes and resume support | | Check calibration inputs and expert coverage | [Calibration](calibration/) | Corpus hashes, counts, per-layer/expert coverage gate | | Check that the intended model really works | [Runtime checks](runtime-checks/) | Model-list check, real completion, JSON output, graph-log audit | | Compare inference throughput fairly | [Throughput](throughput/) | Common-window analysis of exact token-emission traces | | Package reproducible evidence | [Evidence](evidence/) | SHA-256 sealing and tamper/missing-file verification | “Intelligence” is measured through specific tasks, not a single invented composite score. DeepSWE is the concrete software benchmark included here; it is not labeled as a separate benchmark called “Deep Suite.” ## Start here Python **3.10+** is required. External tools may require newer Python; the pinned Cerebras REAP checkout requires **Python 3.12+**. ```bash git clone https://github.com/0xSero/model-toolkit.git cd model-toolkit python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt python toolkit.py list ``` Most launchers and GPQA use only the Python standard library. NumPy is needed for KL, pruning plans, and route coverage. Install large GPU libraries and benchmark harnesses in separate environments using each folder's README; their dependencies may conflict. ### First check a running model ```bash export OPENAI_API_KEY='your-local-or-provider-key' python toolkit.py run runtime-checks \ --base-url http://localhost:8000/v1 --model your-served-model \ --output runs/smoke.json ``` ### Then run a small evaluation After accepting GPQA's dataset terms and downloading `gpqa_diamond.csv`: ```bash python toolkit.py run gpqa-diamond \ --csv data/gpqa_diamond.csv --model your-served-model \ --base-url http://localhost:8000/v1 --limit 2 \ --output runs/gpqa-smoke ``` Remove `--limit` for the full 198-question set. Configure sampling, reasoning effort, and output budget explicitly for your comparison. Request failures withhold the aggregate accuracy; output truncations are reported separately. ### Preview an expensive command first ```bash python toolkit.py run terminal-bench \ --model openai/your-served-model --jobs-dir runs/tb21 \ --limit 1 --dry-run python toolkit.py run exl3-quantization \ --input models/source --output models/exl3-4bpw \ --work runs/exl3-work --bits 4 --dry-run ``` The previews do not require the external harness or a GPU. Actual runs do. Terminal-Bench and DeepSWE require a container environment. SciCode executes generated code: run its harness inside a disposable evaluation environment. ## How the compression tools fit together ```text Pinned model + calibration corpus ↓ REAP observations → coverage checks → expert keep plan ↓ ↓ upstream pruning / architecture-specific weight surgery ↓ EXL3 conversion → runtime checks → KL + task benchmarks ↓ sealed evidence ``` Observation formats, expert tensor layouts, and quantization kernels vary by architecture. The tool READMEs state the supported boundary. A keep plan does not modify weights. A completed conversion does not prove runtime support or retained model quality. For KL, compare the quantized model against **pruned BF16** to isolate quantization loss, and separately against **original BF16** to measure total pruning-plus-quantization loss. Keep sampling, prompts, tokenization, and task sets fixed when comparing task scores. ## Existing Hugging Face work - [GLM-5.3 REAP fidelity study](https://huggingface.co/datasets/0xSero/glm-5.3-reap-fidelity-study): criterion sweeps, keep plans, KL reports, and GPQA artifacts. - [REAP calibration data](https://huggingface.co/datasets/0xSero/reap-calibration-data-v1): calibration inputs and collection methodology. Audit constituent sources before claiming a benchmark-clean evaluation. - [Qwen layerwise observations](https://huggingface.co/datasets/0xSero/qwen35-reap-layerwise-observations) and [GLM layerwise observations](https://huggingface.co/datasets/0xSero/glm5-layerwise-reap-observations): prior observation artifacts. - [GLM-5.3 observations](https://huggingface.co/datasets/0xSero/glm-5.3-reap-observations-v1): referenced by the fidelity study, but **not anonymously accessible when checked on 2026-09-10**. No tool here requires access to it. - [Cerebras REAP](https://github.com/CerebrasResearch/reap) and [PR #17](https://github.com/CerebrasResearch/reap/pull/17): underlying observation and pruning work. See [REAP observation](reap-observation/) for the executable path and [provenance](docs/PROVENANCE.md) for the connection to earlier sessions and skills. ## Reproducibility and status `upstreams.json` records exact upstream commits. `python toolkit.py fetch NAME` checks out one of them under `.external/`; it does not install dependencies. Record harness/package versions, model and dataset revisions, hardware, context, sampling, timeouts, and all failed attempts with your run. The included tools are tested with synthetic data, mocked API responses, and CLI checks. This release does **not** claim a fresh GPU conversion, full benchmark, or architecture compatibility test. See [validation](docs/VALIDATION.md). Historical model scores are deliberately not presented as results of this release. Run the local checks: ```bash python -m unittest discover -s tests -v ``` ## Credits and licensing MIT for this toolkit's code; imported MIT notices are retained in `licenses/`. External harnesses, datasets, models, and model-specific code retain their own licenses. REAP belongs to Cerebras Research; ExLlamaV3/EXL3 to turboderp and contributors; Terminal-Bench/Harbor to their authors; DeepSWE/Pier to Datacurve; SciCode to its authors; lm-evaluation-harness to EleutherAI and contributors. See [provenance](docs/PROVENANCE.md).

Chargement du README du dépôt…

Skills dans ce dépôt

Scientifiques des données Développeurs de logiciels Administrateurs de réseaux et de systèmes informatiques Analystes en assurance qualité des logiciels et testeurs

4 catégories métier · 100% classifié