ml-service-rules

name	ml-service-rules
description	Rules for the Python ML microservice (Demucs, MMS-CTC, allin1, Florence-2, CLIP). Trigger whenever the user asks to work on anything in `ml/`, any FastAPI endpoint, model loading/unloading, forced alignment, audio analysis, or asset captioning.

Architecture

4 endpoints, all in ml/app/endpoints/:

• POST /align → Demucs + MMS-CTC, returns {words, vocals_path}
• POST /analyze → allin1, returns {beats, downbeats, sections, vocal_regions, energy_curve, markers}
• POST /caption → Florence-2 + CLIP, returns {tags, caption, clip_embedding}
• POST /embed_query → CLIP text embed only

Model lifecycle

# EVERY endpoint except /embed_query follows this pattern:
@app.post("/align")
def align(req: AlignRequest) -> AlignResponse:
    demucs = load_demucs()
    vocals = separate(demucs, req.audio_path)
    del demucs; torch.cuda.empty_cache()      # MANDATORY

    aligner = load_mms_aligner()
    words = forced_align(aligner, vocals, req.lyrics, lang="pol")
    del aligner; torch.cuda.empty_cache()     # MANDATORY

    return AlignResponse(words=words, vocals_path=vocals)

Exception: CLIP in /embed_query stays warm across calls (tiny, hit often).

Install pin-list (don't change without reason)

# pyproject.toml
python = ">=3.10,<3.11"    # madmom compat
numpy = "==1.23.5"
cython = "==0.29.36"
# madmom via: uv pip install --no-build-isolation git+https://github.com/CPJKU/madmom

Pre-processing

• MP3 → mono 16 kHz WAV before alignment (MMS expects 16k mono)
• Strip [Verse], [Chorus], [Bridge], [Intro], [Outro] tags from lyrics — these are Suno metatags, not sung
• Pass --star_frequency segment and --romanize to MMS aligner — handles melisma and held vowels

Testing

Every endpoint needs:

• Pydantic request/response model tests (pytest)
• One "happy path" integration test with a 10s sample WAV in ml/tests/fixtures/
• VRAM leak check: call endpoint 3x, torch.cuda.memory_allocated() must return to baseline

When to ask user

• "Should this endpoint fail hard on missing CUDA, or fall back to CPU slowly?"
• "Expected audio length for alignment — clip to first 60s or process full file?"
• "Model load log verbosity — loguru INFO or DEBUG default?"