audiogear

name

audiogear

description

Annotate speech/TTS datasets with per-clip quality metrics (MOS, SNR, STOI, pitch, bandwidth, speaking rate, WER/CER, gender/emotion), multi-ASR consensus transcription and speaker labeling using the audiogear pipeline; then filter clips by quality and build listenable HTML QA reports. Use when asked to compute audio dataset metrics, build extended_metadata.csv, transcribe unlabeled audio, select clean TTS training data, or QA a speech dataset.

audiogear — speech-dataset annotation pipeline

audiogear (https://github.com/lIkesimba9/audiogear) computes per-clip features over a speech dataset: reader → metric blocks → CSV writer, configured with Hydra, sharded across GPUs/nodes. This skill drives it from any project.

Copy this file into the consuming project as .claude/skills/audiogear/SKILL.md. Set AUDIOGEAR_HOME below to wherever the audiogear checkout lives.

Setup (once per machine)

export AUDIOGEAR_HOME=~/audiogear            # adjust; add to the project env
git clone https://github.com/lIkesimba9/audiogear.git "$AUDIOGEAR_HOME" || true
cd "$AUDIOGEAR_HOME"
uv sync --extra ru-pipeline                  # or à-la-carte: --extra mos --extra asr --extra pitch
sudo apt-get install -y espeak-ng            # needed only for SpeakingRateMetric
cp -n .env.example .env                      # put HF_TOKEN= in .env for gated models (pyannote)

Python 3.10–3.12. GigaAM pins torch <2.6 — do not "upgrade" torch inside this venv.

Input contract

One directory per dataset with a |-delimited metadata.csv:

<data_root>/<dataset>/
  metadata.csv        # one row per clip
  audio/...           # referenced by audio_path, relative to the dataset dir

Required columns: id (unique), audio_path, text (may be empty). All other columns pass through untouched; computed metric columns are appended. If the source data has a different shape, first write a converter that produces this layout (see build_betterset-style scripts) — do NOT teach audiogear the foreign schema.

Core workflow

1. Write a dataset config

Create $AUDIOGEAR_HOME/configs/feat_<ds>.yaml. Proven template (mirrors the generated feat_* configs) with CPU∥GPU lanes so DSP metrics hide under GPU time:

defaults:
  - reader: ru          # CsvReader preset: delimiter "|", audio_key=audio_path, id_key=id
  - writer: csv
  - executor: local
  - _self_
hydra:
  job: {chdir: false}
  run: {dir: .}
  output_subdir: null
device: cuda
reader:
  data_folder: /path/to/data_root/<ds>
  glob_pattern: metadata.csv
  limit: -1             # set to 10 for a dry run
writer:
  output_folder: outputs/<ds>
  output_filename: ext_$rank.csv
  sep: "|"
executor:
  tasks: 8              # shards; rows are split round-robin
  workers: 2            # concurrent processes; each pins ONE GPU (rank % gpus)
  skip_completed: true
  logging_dir: logs/<ds>
metrics:
  - _target_: audiogear.pipeline.parallel.ParallelLanes
    lanes:
      cpu:
        - _target_: audiogear.pipeline.metrics.bandwidth.BandwidthMetric
        - _target_: audiogear.pipeline.metrics.wada_snr.SnrMetric
        - _target_: audiogear.pipeline.metrics.speaking_rate.SpeakingRateMetric
          language: ru
      gpu:
        - _target_: audiogear.pipeline.metrics.distillmos.DistillMosMetric
          device: ${device}
        - _target_: audiogear.pipeline.metrics.squim.SquimMetrics
          device: ${device}
        - _target_: audiogear.pipeline.metrics.pitch.CrepePitchMetric
          device: ${device}
          crepe_model: tiny
        - _target_: audiogear.pipeline.metrics.style.StyleMetric   # after pitch (reuses pitch_*)
        - _target_: audiogear.pipeline.metrics.gender.GenderMetric
          device: ${device}

Add WhisperWer to the gpu lane only when text is human-authored.

Lane rules (violating them corrupts data silently):

• lanes must write disjoint columns;
• a metric that consumes another's output (StyleMetric ← pitch_mean, WhisperWer/SpeakingRateMetric ← text) goes in the SAME lane, after its producer; a producer needed by both lanes (e.g. ConsensusTranscriber) runs as an ordinary step BEFORE the lanes block.

2. Dry run, then full run

cd "$AUDIOGEAR_HOME"
uv run python process.py --config-name feat_<ds> reader.limit=10   # smoke test, downloads models
uv run python process.py --config-name feat_<ds>                   # full run

Multi-GPU: executor.workers=<n_gpus>. executor.gpus is an int (count), never a list. Never set CUDA_VISIBLE_DEVICES yourself — the executor pins it. Multi-node: launch the same command once per node under SLURM, or set AUDIOGEAR_NODE_RANK/AUDIOGEAR_NUM_NODES; put executor.logging_dir on shared storage.

3. Merge shards → extended_metadata.csv

Each shard writes outputs/<ds>/ext_<rank>.csv. Merge with id-dedup (duplicates appear if executor.tasks changed between an interrupted run and its rerun):

export AUDIOGEAR_DATA_DIR=/path/to/data_root
uv run python examples/run_batch.py <ds>     # run + merge + .FEAT_DONE marker per dataset

For many datasets, examples/gen_configs.py generates the per-dataset configs and examples/run_batch.py sweeps them smallest-first with resume.

4. Filter + QA

uv run python examples/filter_clean.py <ds>   # extended_metadata.csv -> clean_metadata.csv + drop reasons
uv run python examples/qa_report.py <ds>      # -> <data_root>/<ds>/qa_report.html

qa_report.html is self-contained: metric histograms with the filter thresholds drawn in, and playable audio just below/just above each threshold. To calibrate thresholds: listen to both sides — rejected clips sounding fine ⇒ threshold too strict; accepted clips sounding bad ⇒ too loose. Edit TH in examples/filter_clean.py (single source of truth for filter and report) and regenerate. Set HUMAN_TEXT/ASR_TEXT there: the CER filter is only valid where text was written by humans, not by an ASR model.

Metric catalogue

Class (audiogear.pipeline.metrics.*)	Columns	Notes
distillmos.DistillMosMetric	distillmos	no-ref MOS, GPU
squim.SquimMetrics	pyt_stoi, pyt_pesq, pyt_si_sdr	GPU, never batched (padding skews scores)
wada_snr.SnrMetric	wada_snr	blind SNR, CPU
bandwidth.BandwidthMetric	bandwidth_hz, is_upsampled_est	detects upsampled audio; container sample_rate lies
pitch.CrepePitchMetric / pitch.PitchMetric	pitch_mean, pitch_std	GPU torchcrepe / CPU pyin
style.StyleMetric	energy_db, energy_dynamics, expressiveness	CPU; reuses pitch_* if present
speaking_rate.SpeakingRateMetric	speaking_rate, phonemes_per_word, char_rate	espeak-ng; NOT thread-safe — leave it serial
wer.WhisperWer	whisper_wer, whisper_cer	vs text; -1 when no reference. Skip for ASR-derived text (CER≈0, meaningless, expensive)
gigaam_v3.GigaAMv3	gigaam3_text, gigaam3_cer, text_punctuated, opt-in words JSON	punctuated-from-audio RU transcript + punctuation transferred onto reference words; model_name: multilingual_ctc for ru/en/kk/ky/uz (no punctuation — set punct_column: null); words_column adds word timestamps
gender.GenderMetric / emotion.EmotionMetric	gender_pred / emotion_pred, emotion_score	_pred suffix — never overwrite curated columns
hf.HFAudioModelMetric	configurable	any 🤗 audio classification/regression model, config-only
brouhaha_snr_reverb.SnrReverbMetrics	snr, c50	pyannote, gated (HF_TOKEN)
speaker.SpeakerLabeler	speaker, speaker_conf, speaker_margin	dataset-level clustering — run with executor.tasks=1
transcribers.consensus.ConsensusTranscriber	text, asr_text_*, asr_agreement	multi-ASR medoid; for datasets without transcripts

Reliability semantics (what to expect downstream)

• Sentinels, not crashes. A corrupt/unscorable clip never kills a run: the per-clip guard writes NaN (or -1 for WER/CER/SQUIM) and continues. Always treat NaN and negative metric values as "no value" when filtering. A long unbroken failure streak (default 50) aborts the shard — that means a systematic problem (model/config), not bad data.
• Resume is two-level. Completed shards are skipped via markers in logging_dir; inside an unfinished shard, per-metric JSONL checkpoints in <output_folder>/checkpoints/ resume from the last computed clip. Rerunning the same command after any crash is always safe. Checkpoints are keyed by clip id only — after changing a metric's model/parameters, delete <output_folder>/checkpoints/ (or the run reuses stale values). Disable with resume: false in the config.

Pitfalls checklist

• executor.gpus: 1 (int) — gpus: [0] breaks.
• Whisper compute_type: int8_float16 when 2 workers share a GPU (halves VRAM).
• Clips >25 s are truncated by GigaAMv3; SQUIM needs ≥1024 samples (else -1).
• Audio must be locally readable (torchaudio); CSVs/outputs/logs may be s3://… (fsspec), audio may not — FUSE-mount buckets instead.
• Models load lazily per worker process — first clips are slow, that's normal.
• Adding a metric: subclass BaseMetric, implement compute_metric(segment), set metric= to the column name(s). Do NOT add try/except around it — the base class guards every path already. Load models lazily via audiogear.utils.runtime.cached_model.

Programmatic use (no CLI)

from omegaconf import OmegaConf
from audiogear.build import build_pipeline

cfg = OmegaConf.create({...})       # same shape as a yaml config
data = None
for step in build_pipeline(cfg):    # reader -> metrics -> writer
    data = step(data, 0, 1)         # (data, rank, world_size)
# data: list[AudioSegment]; computed values in segment.metadata