libsync-build-and-env

name

libsync-build-and-env

description

Use when setting up libsync from a fresh clone; when make deps, poetry install, or npm install fails; when ModuleNotFoundError: No module named 'qobuz' or 'tidal' appears; when docker build fails or bakes unexpected SDK code; when make dev serves a 404 UI or make dev-frontend requests hit the wrong port; when the sdks/qobuz_api_client submodule is empty, dirty, or needs a pin bump; or when local Python/Node/Poetry versions differ from CI and behavior diverges.

libsync: build and environment runbook

This skill recreates the libsync development environment from scratch and documents every known build trap. All paths are relative to the repository root. Facts marked with a date were verified on that date against the repo; re-verification commands are at the bottom.

1. From clone to running

Definitions used throughout:

• Poetry: the Python dependency manager that owns the backend virtualenv (pyproject.toml + poetry.lock).
• The submodule: the git submodule at sdks/qobuz_api_client, which contains both the qobuz and tidal Python SDK packages.
• Editable install: pip install -e <path>, which links a package into the virtualenv from a source directory instead of copying it.

Run these commands in order:

git clone --recursive [e-mail]:arthursoares/libsync.git
cd libsync
make deps                          # submodule init + editable-install both SDKs
poetry install                     # backend dependencies into the Poetry venv
cd frontend && npm install && cd ..  # frontend dependencies
make dev                           # build frontend, serve everything at http://localhost:8080

make deps does exactly three things (Makefile, as of 2026-07-03):

git submodule update --init --recursive
poetry run pip install -e sdks/qobuz_api_client/clients/python        # package "qobuz"
poetry run pip install -e sdks/qobuz_api_client/clients/python/tidal  # package "tidal"

The SDKs are not Poetry dependencies. They are editable pip installs into the Poetry venv. pyproject.toml says this explicitly in a comment; the SDKs carry their own dependencies (mutagen, aiolimiter, pycryptodomex, aiofiles).

Install-order note: the README runs make deps before poetry install; CI (.github/workflows/pytest.yml) runs poetry install first and then the two SDK pip installs. Both orders work because poetry run auto-creates the venv on first use. Do not "fix" one to match the other without reason.

Smoke checks after setup:

make test                                          # expect: 154 passed, ~2-3s pytest time (as of 2026-07-03, v0.0.6)
poetry run python -c "import qobuz, tidal; print('SDKs OK')"
which ffmpeg                                       # see section 2 if empty

If make test floods the terminal with DEBUG log blocks, that is configured behavior (log_cli = true, log_level = DEBUG in pyproject.toml), not a failure.

2. Toolchain matrix

Tool	CI / Docker pin	Local constraint in repo	Version file?
Python	3.12 (pytest.yml, docker/Dockerfile python:3.12-slim)	pyproject.toml: >=3.10 <4.0	No .python-version
Node	20 (pytest.yml, node:20-alpine)	README says "Node 20+"; no engines field in frontend/package.json	No .nvmrc
Poetry	1.8.0 (snok/install-poetry@v1 with version: 1.8.0)	none	none
ffmpeg	apt-installed in Docker only	README lists it as a local requirement; nothing installs it	n/a

Traps in this matrix:

• Known inconsistency (as of 2026-07-03): the CI Poetry pin looks stale. poetry.lock declares lock-version = "2.1" and its header says "generated by Poetry 2.3.2" — that is the Poetry 2.x lockfile format, while CI pins Poetry 1.8.0. CI is currently green, so either the action resolves differently or 1.8.0 tolerates the lock; this has not been confirmed either way. If CI dependency installs start failing mysteriously, suspect this first. Do not regenerate poetry.lock with a different Poetry major version as a drive-by.
• Local machines drift freely. Because no .python-version / .nvmrc / .tool-versions exists, your local Python/Node/Poetry can be years newer than CI. pyproject.toml allows any Python >=3.10 <4.0, so this usually works, but when local and CI results disagree, check versions before debugging code.
• ffmpeg is a silent runtime dependency. The Tidal SDK downloader calls shutil.which("ffmpeg") to remux DASH-delivered FLAC-in-MP4 into real .flac files (sdks/qobuz_api_client/clients/python/tidal/tidal/downloader.py). If ffmpeg is missing, downloads do not fail — files are silently left as MP4-wrapped FLAC in .flac-named files, which strict FLAC tools choke on. Nothing in make deps or poetry install installs it. Check with which ffmpeg; install with brew install ffmpeg on macOS. Docker images already include it.

3. The submodule (sdks/qobuz_api_client)

• What it is: a git submodule pinned by gitlink (a commit hash recorded in the parent repo's tree) to https://github.com/arthursoares/qobuz_tidal_api_client.git. The path sdks/qobuz_api_client deliberately keeps the old repo name after the upstream repo was renamed to qobuz_tidal_api_client — do not rename the path.
• Current pin: commit 5e66211 ("feat(tidal): PKCE OAuth + DASH manifest support + folder label fixes") (as of 2026-07-03, v0.0.6).
• Update procedure (per CLAUDE.md): git -C sdks/qobuz_api_client pull, re-run make deps, run make test, then commit the submodule bump in the parent repo through the normal PR flow.

Failure modes:

Symptom	Cause	Fix
make deps pip installs fail; sdks/qobuz_api_client/ is empty	Cloned without --recursive, submodule never initialized	git submodule update --init --recursive, then re-run make deps
docker build fails at the COPY sdks/... step, or the image lacks SDKs	Same — the Dockerfile COPYs the submodule from the host checkout; there is no in-Dockerfile clone fallback (the Dockerfile comment says so)	Initialize the submodule on the host first
ModuleNotFoundError: No module named 'qobuz' (or 'tidal') in pytest or at runtime	Editable installs are invisible to Poetry: poetry sync (Poetry 2.x), poetry install --sync, or any venv rebuild silently removes them	Re-run make deps
make deps fails on a checkout of an old libsync commit — the pinned SDK commit does not exist upstream	The SDK repo's history has been rewritten; the parent repo records at least one re-pin after a rewrite (commit 52837cd "re-pin qobuz_api_client submodule after SDK history rewrite") plus a rename/make-public bump (89e8599). Old gitlinks may point at commits that no longer exist upstream.	Check out a libsync commit whose pin exists, or fetch the SDK repo's current head and accept the skew

Rule of thumb: after any operation that touches the Poetry venv, re-run make deps. It is idempotent and cheap.

4. Makefile target anatomy

Every target, verified against the Makefile (as of 2026-07-03):

Target	What it actually does	Notes
deps	Submodule init + two editable SDK installs	See section 1
test	poetry run pytest tests/ -q --tb=short	154 tests, no credentials needed
test-unit / test-all	Same as test but -v	Identical to each other; both verbose
build-frontend	cd frontend && npm run build	Output goes to frontend/build
build-local	build-frontend, then rm -rf backend/static && cp -r frontend/build backend/static	Hazard: if the frontend build fails, backend/static may already be deleted. The backend only mounts static files when the directory exists (backend/main.py, static-serving block), so the API still runs but every UI route 404s. Fix: re-run make build-local after fixing the build error.
dev	build-local, mkdir -p data, then uvicorn on :8080 with --factory --reload and env vars STREAMRIP_DB_PATH=data/streamrip.db STREAMRIP_DOWNLOADS_PATH=~/Downloads/Music	The primary dev loop
dev-backend	Same uvicorn line, no frontend rebuild	Use when only touching backend code
dev-frontend	cd frontend && npm run dev (Vite dev server on :5173)	Broken by design gap — see below
build-docker	docker build -f docker/Dockerfile -t streamrip .	Image tag keeps the legacy name deliberately
run-docker	Runs the image with ~/Downloads/Music:/music, named volume streamrip-data:/data, STREAMRIP_DB_PATH=/data/streamrip.db
docker	build-docker + run-docker
lint	poetry run ruff check backend/	Currently FAILS on clean main due to local-vs-CI ruff version skew — do not blindly --fix. Details live in the sibling skill libsync-validation-and-qa.

make dev-frontend does not work out of the box (as of 2026-07-03). The frontend API client hardcodes BASE = '/api' with relative fetches (frontend/src/lib/api/client.ts), and frontend/vite.config.ts contains no proxy configuration — it is just defineConfig({ plugins: [sveltekit()] }). So API requests from the Vite dev server hit :5173 instead of the backend on :8080, and there is no CORS setup either. The Makefile comment "proxied to a running backend" and the matching README line are stale. The reliable loop is make dev (full) or make dev-backend (backend-only, reusing the last-built backend/static). Whether a Vite proxy is planned is an open question — treat any fix as a change needing the normal PR flow.

Environment variables the backend reads — exactly two (verified by grepping backend/ for os.environ/os.getenv, as of 2026-07-03): STREAMRIP_DB_PATH (default data/streamrip.db) and STREAMRIP_DOWNLOADS_PATH (default /music). Both are only fallbacks: a downloads_path value in the SQLite config table overrides the env var. The per-source dedup DBs are derived from STREAMRIP_DB_PATH's directory: downloads.db for Qobuz (legacy name kept on purpose), downloads-<source>.db for others (backend/services/download.py, _download_album area). Full config catalog: see sibling skill libsync-config-and-flags.

5. Docker build

docker/Dockerfile is a two-stage build (as of 2026-07-03):

1. Stage 1 (node:20-alpine): npm ci from frontend/package*.json, then npm run build.
2. Stage 2 (python:3.12-slim): apt-installs ffmpeg; copies pyproject.toml, poetry.lock, README.md, backend/, and sdks/qobuz_api_client/clients/python/ (renamed inside the image to ./sdks/qobuz_api_client_py/); runs pip install poetry, poetry config virtualenvs.create false, poetry install --only main, then pip install websockets ./sdks/qobuz_api_client_py ./sdks/qobuz_api_client_py/tidal; copies the stage-1 frontend build into ./backend/static/. Exposes 8080; CMD is uvicorn with --factory.

Traps:

• The build is not fully lockfile-driven. poetry install --only main uses poetry.lock, but the follow-up pip install websockets is unpinned, and the SDKs are installed from whatever the host's submodule checkout contains at build time.
• A local docker build bakes in uncommitted submodule edits. The COPY takes the submodule working tree as-is, including local modifications (see section 6 — the worktree is currently dirty). If you need a reproducible image, verify git -C sdks/qobuz_api_client status --short is clean first, or use the CI-published image from GHCR.
• Uninitialized submodule = broken build. Same as section 3: initialize on the host before docker build.
• Two compose files exist: docker/docker-compose.yml (builds locally, named volume streamrip-data) and docker-compose.example.yml at repo root (pulls ghcr.io/arthursoares/libsync:latest, bind-mounts ./music and ./data). Running and operating containers is covered by sibling skill libsync-run-and-operate.

CI's docker-publish job is a required merge check — a Dockerfile breakage blocks all merges to main. Never route around this; fix the build.

6. Known local-state facts (dated 2026-07-03 — verify before relying on them)

• The SDK submodule working tree carries 31 modified files (git -C sdks/qobuz_api_client status --short shows 31 M entries, +65/−80 lines) while the gitlink pin itself is clean at 5e66211. Spot-checked diffs are lint-only churn (import re-sorting, f-string→plain-string); not every file was audited. Because the SDKs are editable installs, these uncommitted edits ARE live in the running backend and would be baked into a local docker build. Do not reset or commit these files without maintainer say-so (inferred — confirm with maintainer; they may be in-flight upstream work). The parent-repo git status shows this as m sdks/qobuz_api_client (lowercase m = dirty content, not a changed pin).
• data/ contains legacy zombie DBs. Current contents: streamrip.db (the live config/library DB), downloads.db (live Qobuz dedup DB), plus downloads_albums.db and failed.db, which no code in backend/ or tests/ references (verified by grep) — they are leftovers from the deleted CLI era. Deleting data/ entirely wipes local credentials and library state; what is safe to delete is the domain of sibling skill libsync-run-and-operate.
• data/ is gitignored (both data/ and *.db patterns in .gitignore), so none of this can leak into commits.

When NOT to use this skill

• Running, deploying, releasing, or deciding what data artifacts are safe to delete → libsync-run-and-operate.
• Lint/format version-skew details, pre-push checklist, what CI actually gates → libsync-validation-and-qa.
• Config keys, env-var precedence, and DB-config overrides in depth → libsync-config-and-flags.
• A build that used to work and now fails for non-environment reasons → libsync-debugging-playbook.
• How to get a change (e.g. a Vite proxy fix, a Poetry pin bump) merged → libsync-change-control.
• SDK internals, auth flows, quality tiers → qobuz-tidal-domain-reference.

Provenance and maintenance

Re-verify each drift-prone fact before trusting it:

• make deps contents and all Makefile targets: cat Makefile
• README setup order and requirements: sed -n '24,100p' README.md
• CI pins (Python 3.12, Poetry 1.8.0, Node 20): grep -n "python-version\|node-version\|version:" .github/workflows/pytest.yml
• poetry.lock format vs CI Poetry pin: head -1 poetry.lock && grep -n "lock-version" poetry.lock
• Python constraint and SDK-install comment: grep -n "python =" pyproject.toml && sed -n '22,27p' pyproject.toml
• Absence of version files: ls .python-version .nvmrc .tool-versions frontend/.nvmrc 2>&1
• No engines field: grep -n engines frontend/package.json || echo "no engines field"
• Submodule pin and remote: git submodule status && cat .gitmodules
• Submodule worktree dirtiness: git -C sdks/qobuz_api_client status --short | wc -l
• SDKs still importable in the venv: poetry run python -c "import qobuz, tidal; print('OK')"
• Test count/time: time make test
• Dockerfile anatomy (COPY-from-host, unpinned websockets): cat docker/Dockerfile
• ffmpeg presence and its use in the Tidal SDK: which ffmpeg; grep -n 'shutil.which("ffmpeg")' sdks/qobuz_api_client/clients/python/tidal/tidal/downloader.py
• No Vite proxy / hardcoded /api base: cat frontend/vite.config.ts && grep -n "BASE" frontend/src/lib/api/client.ts
• Backend env vars (should remain exactly two): grep -rn "os.environ\|os.getenv" backend/
• Zombie DBs unreferenced: ls data/ && grep -rn "downloads_albums\|failed.db" backend/ tests/ || echo "unreferenced"