github-setup

name

github-setup

description

Use when bringing up an open-source GitHub research repo (any field) — cloning, configuring conda env, installing dependencies, downloading weights, running the demo. Triggering signals — README has a `quick_install.sh` / `requirements.txt` and you're about to follow it; pip install hangs / fails / pulls a different numpy unexpectedly; `import X` works but the demo crashes; demo "should take 30s" but takes hours.

GitHub Repo Setup — Generic Workflow

Overview

Open-source research repos almost always have brittle install scripts: README pip pins were tested on the author's machine + the day they merged it, then drift. Following the README literally usually fails. This skill captures the generic workflow — what to do in what order, regardless of the repo — and supplies a real case study (pixel3dmm + SMIRK) showing how each generic principle landed concretely.

Core principle: install the bare minimum, verify imports, smoke-test on a tiny sample, then fan out. Don't trust pip pin combinations blindly; numpy / cv2 / cuda transitive deps shift under your feet.

When to Use

• Cloning any GitHub research repo and following its install instructions
• Adding a sister project to an existing conda env (only diff packages)
• Running a demo that depends on heavy weights (FLAME, SMPL-X, mediapipe tasks, gdown ckpts)
• A pip install hangs / fails / pulls an unexpected version
• import X works but X.load(ckpt) fails with cryptic error
• A demo "should take 30s" but takes hours — likely env or resource issue

Don't use for: production deployment (use container), pure CPU-only utility libs.

Generic Workflow

digraph workflow {
    "Have working sister env?" [shape=diamond];
    "Reuse + diff install" [shape=box];
    "Fresh conda env" [shape=box];
    "Read README to end" [shape=box];
    "Install in 8 phases" [shape=box];
    "Smoke test on tiny sample" [shape=doublecircle];
    "Fail?" [shape=diamond];
    "Diagnose by category" [shape=box];

    "Have working sister env?" -> "Reuse + diff install" [label="yes"];
    "Have working sister env?" -> "Fresh conda env" [label="no"];
    "Reuse + diff install" -> "Smoke test on tiny sample";
    "Fresh conda env" -> "Read README to end";
    "Read README to end" -> "Install in 8 phases";
    "Install in 8 phases" -> "Smoke test on tiny sample";
    "Smoke test on tiny sample" -> "Fail?";
    "Fail?" -> "Diagnose by category" [label="yes"];
    "Diagnose by category" -> "Install in 8 phases";
    "Fail?" -> "Persist as install_env.sh" [label="no"];
}

The 8 phases

1. Decide: reuse sister env vs fresh env
2. Read README in full: list all weights / sub-repos / version pins / external services (FLAME website, gdown IDs, etc.) BEFORE running anything
3. Harden the env: PATH order, pip index, build-isolation defaults
4. Install torch + CUDA toolkit + build tools: verify torch.cuda.is_available() before touching anything else
5. Install requirements.txt: pin numpy + setuptools first, then --no-build-isolation for legacy or compile-from-source packages
6. Compile-from-source extras (pytorch3d, nvdiffrast, custom CUDA kernels): always --no-build-isolation
7. Sub-repos + heavy weights: gdown / wget; verify file sizes; place in expected paths
8. Re-pin the packages that get bumped by transitive deps (numpy, opencv); add monkey-patches if needed (torch.load weights_only)

Then: smoke test on a tiny sample. If it passes, persist the working commands as one idempotent install_env.sh.

Failure Categories (diagnose by symptom, not by repo)

When something breaks, identify which category the failure belongs to. Each category has a fixed playbook.

Category	Symptom	Playbook
A. Env isolation	which python shows base after conda activate ENV	Manually prepend $ENV/bin:$PATH
B. Build dependency	ModuleNotFoundError during pip build	--no-build-isolation + ensure deps pre-installed
C. Version compatibility matrix	Runtime AttributeError on numpy / cv2 / torch	Identify the conflict pair, pin lower one, re-install
D. Network / mirror	pip / wget timeouts, gdown 403, conda checksum mismatch	Switch index, retry, use mirror, clear cache
E. Asset placement	FileNotFoundError for FLAME / pretrained ckpt	Verify directory structure isn't a placeholder; cp not symlink
F. System-level compatibility	libx264 not found, EGL not initialized	Use conda-forge alternative, accept CPU fallback, swap library
G. API breaking change	Cryptic ckpt-load error, signature mismatch	Monkey-patch defaults at script entry
H. Resource contention	Demo 100× slower than README claims	nvidia-smi, ps aux --sort=-%cpu BEFORE blaming env

Each generic problem maps to a category. Once you recognize the category, the playbook is short.

Reuse Pattern (sister env)

Most projects ride on top of an existing torch-cuda-python stack. Adding only the diff is faster + less risky than fresh install.

source ~/miniconda3/etc/profile.d/conda.sh
conda activate sister_env
export PATH="$CONDA_PREFIX/bin:$PATH"
export PIP_INDEX_URL="https://pypi.org/simple"

# 1. inventory current versions
pip list | grep -iE "torch|numpy|opencv|cuda|matplotlib"

# 2. install ONLY the diff packages from new repo's requirements.txt
pip install --no-build-isolation --no-cache-dir <only_diff_packages>

# 3. immediately re-pin packages that diff installs may have bumped
pip install --no-build-isolation --no-cache-dir "numpy==<known_good>" "opencv-python<<known_good>"

# 4. verify BOTH sister and new repo's smoke tests still pass

Smoke Test Pattern

After ANY install, before doing anything else:

# Pick the smallest sample (5s clip / single image), NOT the full dataset
time python demo.py --input samples/short_clip.mp4 --out /tmp/test.npz

If it fails: STOP, diagnose by category. The 8 categories cover most of what you'll see.

If it succeeds but is 100× slower than README claims: check resource contention before blaming env (nvidia-smi, ps aux). Background training / inference jobs can starve your demo silently.

Common Mistakes

Trusting README pin combos: numpy==1.22.4 + opencv==4.9 + mediapipe==0.10.10 may pull numpy 2.0 transitively. Always re-pin numpy AFTER the full install (Category C).

Skipping smoke test: "I'll just run the full dataset" → 6 hours later you find out step 1 was misconfigured.

Trusting partial git clone state: previous failed install left half a checkout. New install thinks repo exists, skips clone, files missing. Always ls subrepo/ not just [ -d subrepo ].

5 retry scripts instead of 1 fixed script: each retry should patch the original script (idempotent guards), not become install_v2.sh / install_v3.sh / ....

Not isolating GPU contention: 100× slower than expected → check nvidia-smi BEFORE debugging env. Background training jobs are silent thieves.

Red Flags — STOP and Diagnose

• pip output Successfully installed numpy-2.0.x after you pinned 1.23.x → re-pin (Category C)
• mediapipe first-frame call > 5s → fell back to CPU (Category F)
• Demo "completes" but output file empty / wrong shape → smoke test caught a logic bug, not env
• README says "30 seconds on a single GPU" but you see 30 minutes → Category H (resource) or F (CPU fallback)
• conda activate X then which python shows base path → Category A
• Three install_*_resume.sh files in the project → you're not idempotent; consolidate

Persistence

After the smoke test passes, commit the working install as ONE script:

install_env.sh

Requirements:

• set -euxo pipefail so failures stop loud
• [ -d X ] || git clone ... guards so re-running is safe
• All 8 phases inline, with which Category each step addresses commented
• Move all _v2.sh / _resume.sh retry artifacts to legacy/

Case Study: Pixel3DMM + SMIRK (concrete pitfalls hit in this project)

This is one repo's worth of concrete examples for the categories above. Use as reference for what these problems actually look like in the wild.

#	What happened	Category	Fix
1	conda activate p3dmm then which python shows base 3.13 instead of env's 3.9	A	export PATH="$CONDA_PREFIX/bin:$PATH" after activate
2	pip stuck on Tsinghua mirror read timed out	D	export PIP_INDEX_URL="https://pypi.org/simple"
3	chumpy setup.py imports pip (legacy code) → ModuleNotFoundError in build env	B	pip install --no-build-isolation chumpy
4	pytorch3d source build can't find torch (build isolation creates fresh env)	B	pip install --no-build-isolation git+...
5	conda CUDA tarball ChecksumMismatchError (libcusolver)	D	mv $CONDA_PKGS/*.tar.bz2 /tmp/, retry
6	ln -sfn /path/FLAME2020 ./MICA/data/FLAME2020 ends up nested (target dir already exists as placeholder)	E	cp /path/FLAME2020/*.pkl ./MICA/data/FLAME2020/
7	PIPNet's .pyx files use np.int_t (removed since numpy ≥1.20), Cython 3 doesn't compile	C	sed -i 's/np\.int_t/np.intp_t/g' *.pyx; pip install "Cython<3"
8	Runtime AttributeError: module 'numpy' has no attribute 'int' (mediapipe / opencv 4.13 pulled numpy 2.0)	C	pip install "numpy==1.23.5" "opencv-python<4.10" after every install that may bump numpy
9	torch.load(ckpt) fails: UnpicklingError ... Unsupported global: omegaconf.dictconfig.DictConfig (PyTorch 2.6+ default weights_only=True)	G	Monkey-patch at script top: torch.load = lambda *a, **kw: _orig(*a, **{**kw, "weights_only": False})
10	System ffmpeg unknown -crf / no libx264 (RHEL 8 ffmpeg 3.1)	F	mpeg4 -q:v 1 for high quality OR install conda-forge ffmpeg
11	mediapipe Linux GPU init fails: EGL not initialized	F	Accept CPU fallback (~50-200 ms/frame) OR swap to face_alignment torch GPU (~63 ms/frame)
12	gdown 403 on large Google Drive files	D	gdown <ID> --fuzzy or HuggingFace mirror if available
13	SMIRK demo 53 min for 150 frames despite GPU available	H	Background training had GPU at 100% — wait or use different GPU

Real-world impact of applying the workflow

Pixel3DMM full setup: avoidable 4+ hours of debugging → 30 min with this skill. SMIRK reusing pixel3dmm env: 2 diff packages, 5 min total install. 5s smoke test caught 9/13 pitfalls before processing 89-clip dataset. Final SMIRK inference: 2.6s / 150 frames vs the README's mediapipe path which would have been 30+ s.

See pixel3dmm_test/shell/install_env.sh for the consolidated single-script outcome (vs the 5 retry scripts in legacy/).