freeshard-build-and-env

name

freeshard-build-and-env

description

Set up and verify a working shard_core development environment. Use when starting from a fresh clone or bare machine, creating a git worktree for an issue, running uv sync, wondering what a justfile recipe does (run-dev, cleanup, get-types, set-version, run-from-backup), reading the Dockerfile, or hitting env-shaped failures (import errors, "config.toml not found", pytest can't bind port 5433, docker network 'portal' already exists, black/ruff not found). TRIGGER on: "set up the repo", "uv sync", ".worktrees", "justfile", "just cleanup", "Dockerfile", "CI installs differently than local", fresh-environment smoke test. SKIP when: actually running the shard or the compose stack, dev server usage, backup/restore ops (use freeshard-run-and-operate); writing or debugging tests and fixtures (use freeshard-testing-and-qa); type-sync policy and controller drift details (use freeshard-ecosystem-contracts); config keys and env-override semantics (use freeshard-config-and-flags).

Freeshard build & environment

Goal: from a bare Linux box (or a fresh worktree) to a state where pytest collects, ruff passes, and you can trust your results. Repo = shard_core, the FastAPI backend that runs on each Freeshard VM ("shard") and orchestrates user apps via Docker Compose behind Traefik.

All commands assume CWD = repo root (/home/ubuntu/projects/freeshard/freeshard on the main dev VM; adjust for your clone/worktree).

When NOT to use this skill

You want to...	Use instead
Run the dev server, bring up the compose stack, first-start pairing, backup/restore, release	freeshard-run-and-operate
Understand/extend the test suite, fixtures, flaky tests	freeshard-testing-and-qa
Sync types from the controller, cross-repo contracts, drift checks	freeshard-ecosystem-contracts
Look up a config key, env override rules, dead config	freeshard-config-and-flags
Know what changes are allowed and how they ship	freeshard-change-control

Prerequisites

Tool	Required	On the main dev VM as of 2026-07-03	Check
Python	>= 3.13 (pyproject.toml:15)	3.13.3	python3 --version
uv	any recent	0.10.7	uv --version
just	any recent	1.40.0	just --version
Docker daemon	running, socket accessible	29.2.1	docker info >/dev/null && echo ok
Docker Compose	v2 plugin (docker compose, not docker-compose)	v5.0.2	docker compose version
gh	authenticated	2.46.0	gh auth status
Internet	yes — tests do a real docker login + real image pulls	—	—

uv can provision Python itself (uv python install 3.13) if the system Python is too old. Docker + internet are needed even for "just running the tests": the test suite starts a real postgres:17 container and logs into a real container registry (see traps below).

From bare Linux to productive

git clone https://github.com/FreeshardBase/freeshard.git
cd freeshard
uv sync --extra dev            # creates .venv from uv.lock; --extra dev pulls pytest, ruff, etc.
.venv/bin/pytest tests --collect-only -q   # expect "137 tests collected" (origin/main as of 2026-07-03; 134 on the older e55ce51 checkout — count drifts, cross-check freeshard-testing-and-qa), zero errors
.venv/bin/ruff check .         # expect no findings on a clean main

Notes:

• uv sync installs from uv.lock — the single source of truth for dependency versions. Never pip install . into this venv.
• Dev tools live in the optional extra [project.optional-dependencies].dev (pyproject.toml:47-59). Without --extra dev you get no pytest/ruff.
• black is not listed in the dev extra, but just cleanup calls .venv/bin/black. It works anyway because black is a transitive dependency of datamodel-code-generator (see uv.lock). If just cleanup ever reports black missing, you ran uv sync without --extra dev.
• The shard_core.egg-info/ directory you may see at repo root is a setuptools build artifact, gitignored — ignore it.

Worktree discipline (mandatory for new work)

Convention in this repo: every feature/bugfix gets its own worktree under .worktrees/<slug> on its own branch, cut from freshly pulled origin/main. The overnight agent loop assumes this; parallel sessions must not share a checkout.

git fetch origin main
git worktree add .worktrees/<slug> -b <branch-name> origin/main
cd .worktrees/<slug>
uv sync --extra dev            # ALWAYS, before anything else
.venv/bin/pytest tests --collect-only -q   # sanity: env works

Rules:

• ALWAYS run uv sync --extra dev inside the new worktree. Never symlink or share .venv from the main checkout: branches can diverge on pyproject.toml/uv.lock, and a borrowed venv gives silently wrong test results and pollutes the other checkout with __pycache__ writes.
• .worktrees/ is ignored via .git/info/exclude (local-only; not committed in .gitignore). In a fresh clone, add it yourself: echo '.worktrees/' >> .git/info/exclude.
• Two worktrees cannot run the test suite at the same time — see traps (port 5433, compose project name, docker network portal are all host-global).
• local_config.toml is committed, so every worktree has it — but it is resolved relative to CWD (see traps). Run everything from the worktree root.

Repo layout orientation map

freeshard/                     repo root (package name: shard_core)
├── shard_core/                the Python package
│   ├── app.py                 entrypoint (target of `fastapi run/dev`)
│   ├── app_factory.py         create_app() + lifespan = startup/shutdown order
│   ├── settings.py            pydantic-settings: config.toml + local_config.toml + FREESHARD_* env
│   ├── web/                   FastAPI routers by auth level: public/ protected/ internal/ management/
│   ├── service/               business logic (app_installation/, backup.py, crypto.py, ...)
│   ├── database/              psycopg3 async access layer (yoyo migrations applied at startup)
│   ├── data_model/            pydantic models; backend/ = COPIED from controller — never hand-edit
│   └── util/
├── data/                      runtime templates: traefik.yml(+_no_ssl), welcome-log jinja, splash.html, EFF wordlist
├── migrations/                yoyo SQL migrations (shard-core-0001-init.sql)
├── tests/                     pytest suite; tests/config.toml = overlay; tests/docker-compose.yml = postgres:17 on :5433
├── scripts/                   generate_traefik_dyn_config_model.py — raises on purpose, do NOT run (hand-patched output)
├── config.toml                base config, committed (contains live ACR pull credentials — do not "clean up")
├── local_config.toml          dev overlay: path_root="run/", dns.zone="localhost" (CWD-relative, dockerignored)
├── docker-compose.yml         production stack: postgres + traefik + shard_core + web-terminal on network 'portal'
├── Dockerfile                 two-stage uv image build (see anatomy below)
├── justfile                   task recipes (see anatomy below)
├── uv.lock                    dependency source of truth (CI and image both build --frozen from it)
└── .worktrees/                per-task worktrees (ignored via .git/info/exclude)

justfile recipe anatomy

Verified against justfile as of 2026-07-03. just (no args) lists recipes.

Recipe	What it actually does	Gotchas
just run-dev	fastapi dev --port 8080 shard_core/app.py via .venv/bin/fastapi, with PYTHONUNBUFFERED=1	Config comes from config.toml plus local_config.toml if present, picked up by settings.py automatically. Usage details: freeshard-run-and-operate.
just run-dev-for-freeshard-controller	second dev instance on port 8081, intended to point at a local controller on 8080	Sets FREESHARD_FREESHARD_CONTROLLER__BASE_URL=http://127.0.0.1:8080. Until issue #128 this was a single underscore, which pydantic-settings silently ignored — the instance talked to the production controller. See freeshard-config-and-flags.
just cleanup	ruff check . --fix then black shard_core and black tests	Run after every code change — it is the house formatting gate. Requires the dev extra installed (black is transitive).
just get-types	DESTRUCTIVE: rm -rf shard_core/data_model/backend, then cp -r from the LOCAL sibling checkout ../freeshard-controller/freeshard-controller-backend/freeshard_controller/data_model, prepending # DO NOT MODIFY to every file	Syncs from whatever your local controller checkout happens to be at. You MUST git -C ../freeshard-controller checkout main && git -C ../freeshard-controller pull first. A stale checkout has caused real bugs before (silently dropped billing fields — commits 6d4b101, e55ce51). As of 2026-07-03 the local controller checkout on the dev VM was verified BEHIND origin/main. Full procedure and drift checks: freeshard-ecosystem-contracts.
just set-version X.Y.Z	rewrites version in pyproject.toml and the ghcr.io/freeshardbase/freeshard: tag in docker-compose.yml, then git add . and git commit	It stages EVERYTHING (git add .) — never run with a dirty tree. Release procedure: freeshard-change-control / freeshard-run-and-operate.
just run-from-backup <zip>	DESTRUCTIVE: rm -rf run/ then unzips the backup into run/ (the dev path_root)	Wipes local dev state without asking. Restore workflow: freeshard-run-and-operate.

Dockerfile anatomy (why the image looks like that)

Two-stage build, verified against Dockerfile as of 2026-07-03:

1. Build stage — ghcr.io/astral-sh/uv:python3.13-bookworm-slim. Runs uv sync --frozen --no-install-project --no-dev first (dependencies only, cached layer), then ADD . . + uv sync --frozen --no-dev (project itself). UV_COMPILE_BYTECODE=1, uv cache mounts. --frozen = build exactly from uv.lock.
2. Runtime stage — python:3.13-slim-bookworm plus:
   • curl — used by the container HEALTHCHECK (curl -f http://localhost/public/health).
   • rclone — the backup engine (rclone crypt → Azure Blob).
   • tini as PID 1 (ENTRYPOINT ["/usr/bin/tini","--"]) — reaps the zombie processes the healthcheck curl otherwise leaves behind (https://github.com/FreeshardBase/freeshard/pull/55).
   • Full Docker install via get.docker.com — shard_core drives its sibling app containers by shelling out to docker/docker compose against the docker.sock mounted from the host (docker-compose.yml mounts /var/run/docker.sock). Only the CLI + compose plugin are used; the daemon inside the image never runs.
   • CMD ["fastapi","run","--port","80","shard_core/app.py"].

CI parity rule

Install from the lockfile, exactly like CI and the image do. As of 2026-07-03, .github/workflows/test.yml on origin/main installs with uv sync --frozen --extra dev and runs uv run ruff check . / uv run pytest tests (changed in https://github.com/FreeshardBase/freeshard/pull/114). Before that, CI used pip install ".[dev]", which ignores uv.lock and resolves latest-at-runtime — a fresh upstream release broke the controller's CI that way with zero code changes, and this repo switched preemptively. If you see red CI with no plausible connection to the diff, first check whether the install step still honors the lock.

Match CI locally:

uv sync --frozen --extra dev   # exactly what CI installs
uv run ruff check .            # CI lint job
uv run pytest tests            # CI test job (needs Docker + internet)

uv sync without --frozen will also update uv.lock if pyproject.toml changed — fine during development, but a lockfile diff you didn't intend belongs out of your PR.

One workflow variable is dead but still present (do not cargo-cult it): TEST_ENV=full/sparse — its consumer was removed, so push and PR runs are identical. A companion dead CONFIG=tests/config.yml was removed by issue #128.

Known traps

Trap	Reality	Evidence
local_config.toml silently not applied	It is resolved via Path("local_config.toml").exists() — relative to CWD. Run the server/tests from the repo (or worktree) root, or your dev overlay (path_root=run/, dns.zone=localhost) vanishes and the app tries production paths. Also listed in .dockerignore, so it can never leak into the image — don't try to configure the container with it.	settings.py:148-152; .dockerignore
Tests "just need pytest"	They need a running Docker daemon, compose v2, and internet: a session fixture does a real docker login to portalapps.azurecr.io using credentials committed in config.toml:30-33, and app-installation tests really pull images from that registry. If those credentials are ever rotated, api_client-based tests fail on pulls — that is an infra failure, not your bug.	tests/conftest.py:129-131
Concurrent test runs collide	tests/docker-compose.yml pins host port 5433 and conftest pins compose project name shard-core-test. Two worktrees running pytest simultaneously fight over both. Run suites serially across worktrees.	tests/docker-compose.yml:9; tests/conftest.py:96-98
Docker network portal and container names are host-global	The production compose stack, a locally running dev shard, and the test suite all use the network name portal and fixed container names (postgres, traefik, shard_core, app names like filebrowser). Test teardown force-removes the portal network — it will rip it out from under a dev shard running on the same host.	docker-compose.yml:1-3; tests/util.py:139-147
scripts/generate_traefik_dyn_config_model.py	Raises Exception("Do no use! ...") at import, deliberately: its output model was hand-patched (authResponseHeadersRegex) and regenerating would drop that field. Do not "fix" the script.	scripts/generate_traefik_dyn_config_model.py
Leftover test infra after a crash	A killed pytest run leaves the test postgres and/or portal network behind; the next run fails on port/name conflicts.	Cleanup: docker compose -p shard-core-test -f tests/docker-compose.yml down -v and docker network rm portal

15-minute smoke checklist (fresh environment)

Run top to bottom; each step has a pass condition. Stop and fix at the first failure.

# 1. Toolchain present (~1 min)
python3 --version && uv --version && just --version && gh auth status
docker info >/dev/null && docker compose version        # daemon up, compose v2

# 2. Clean, current source (~1 min)
git fetch origin main && git status                      # expect: clean tree
git log --oneline -1 origin/main                         # note the tip you're building against

# 3. Env from lockfile (~2 min)
uv sync --frozen --extra dev                             # pass: exits 0, no resolver output

# 4. Package imports (~10 s)
.venv/bin/python -c "import shard_core.app_factory; print('import ok')"
# pass: prints "import ok" (a CryptographyDeprecationWarning about OFB is known noise as of 2026-07-03)

# 5. Test collection (~30 s, no Docker yet)
.venv/bin/pytest tests --collect-only -q | tail -1       # pass: "N tests collected", zero errors (137 on origin/main as of 2026-07-03; the count drifts — what matters is zero collection errors)

# 6. Lint (~10 s)
.venv/bin/ruff check .                                   # pass: "All checks passed!"

# 7. One real test — exercises pytest-docker, postgres:17 spin-up on :5433,
#    internet, and the committed ACR docker login (~3-5 min)
.venv/bin/pytest tests/test_crypto.py -q                 # pass: all green

# 8. Formatting gate works (~30 s)
just cleanup && git status                               # pass: exits 0; tree still clean on untouched main

If step 7 fails at container startup: check for a leftover shard-core-test compose project or occupied port 5433 (traps table). If it fails at docker login or image pull: the committed ACR credential may be rotated — escalate rather than debug your own change.

Provenance and maintenance

Written 2026-07-03 against branch state origin/main @ 0a40684 (local checkout on fix-profile-billing-fields @ e55ce51). Primary sources: justfile, Dockerfile, pyproject.toml, uv.lock, shard_core/settings.py, tests/conftest.py, tests/docker-compose.yml, docker-compose.yml, .github/workflows/test.yml (origin/main), README.md, https://github.com/FreeshardBase/freeshard/pull/114, https://github.com/FreeshardBase/freeshard/pull/55. Tool versions quoted are the main dev VM's on the same date.

Drift-prone facts — re-verify before trusting:

Fact (as of 2026-07-03)	Re-verify with
Version 0.39.2 / image tag in compose	grep '^version' pyproject.toml && grep 'freeshardbase/freeshard:' docker-compose.yml
137 tests collected (origin/main; 134 at e55ce51)	.venv/bin/pytest tests --collect-only -q | tail -1
CI installs via uv sync --frozen --extra dev	git fetch origin && git show origin/main:.github/workflows/test.yml | grep -A3 'Install dependencies'
dev extra contents / black still transitive	grep -A12 'dev = \[' pyproject.toml
Test postgres port 5433 / project shard-core-test	grep 5433 tests/docker-compose.yml && grep -n 'shard-core-test' tests/conftest.py
ACR creds still committed & logged into	grep -n 'azurecr' config.toml && grep -n 'login_docker_registries' tests/conftest.py shard_core/service/app_installation/__init__.py
Local controller checkout current before get-types	git -C ../freeshard-controller rev-parse HEAD && git -C ../freeshard-controller ls-remote origin main (hashes must match)
.worktrees/ excluded locally	grep worktrees .git/info/exclude
Crypto still RSA-4096/PSS	grep -n 'generate_private_key|key_size|RSA_PSS' shard_core/service/crypto.py shard_core/service/signed_call.py
tini/docker/rclone still in runtime image	grep -n 'tini|get.docker.com|rclone' Dockerfile