upstream-sync

# Upstream Sync Sync content from an upstream repository into the local repo, handling different directory structures via path mapping. ## Bundled Scripts The skill includes sync scripts in `scripts/`: - `scripts/sync-upstream.ts` — 5-phase sync orchestrator - `scripts/sync-utils.ts` — path mapping, git helpers, integrity checks ## Workflow ### Step 0: Deploy Scripts to Project Copy the bundled scripts to the project's `scripts/` directory (skip if already present and up-to-date): ```bash mkdir -p <PROJECT_ROOT>/scripts cp <SKILL_DIR>/scripts/sync-upstream.ts <PROJECT_ROOT>/scripts/ cp <SKILL_DIR>/scripts/sync-utils.ts <PROJECT_ROOT>/scripts/ ``` The project needs `tsx` in devDependencies. If missing: `npm i -D tsx`. ### Step 1: Configure Upstream Remote ```bash git remote get-url upstream 2>/dev/null || git remote add upstream <UPSTREAM_URL> git fetch upstream ``` ### Step 2: Generate Path Mappings Analyze both repos and create `<PROJECT_ROOT>/.upstream-mapping.json`: ```bash # List upstream root structure git ls-tree --name-only upstream/main # List local content directories ls <PROJECT_ROOT>/docs/ ``` Generate mapping config: ```json { "_comment": "Upstream -> local path mapping", "exclude": ["README.md"], "mappings": [ { "upstream": "01_intro/", "local": "docs/zh/01_intro/" }, { "upstream": "02_fundamentals/", "local": "docs/zh/02_fundamentals/" }, { "upstream": "_images/", "local": "docs/zh/_images/" }, { "upstream": "SUMMARY.md", "local": "SUMMARY.md" } ] } ``` **Mapping rules:** - Directory: upstream path ends with `/`, local path ends with `/` - File: exact paths, no trailing `/` - `exclude`: upstream paths skipped entirely (never synced) - First match wins **Preserve local rules (`preserveLocal`):** Use `preserveLocal` to specify line patterns that should be preserved from the local version during sync. This is useful when you've made local customizations (like replacing GitHub URLs with local paths) that should not be overwritten by upstream changes. ```json { "preserveLocal": [ { "description": "Keep local code file links instead of GitHub URLs", "pattern": "\\[code/[^\\]]+\\]\\(code/", "glob": "*.md" }, { "description": "Keep local relative doc links instead of GitHub absolute paths", "pattern": "\\]\\(\\.\\./chapter", "glob": "*.md" } ] } ``` **Preserve rule fields:** - `description`: Human-readable description of what this rule preserves - `pattern`: Regex pattern to match lines that should be preserved from local version - `glob` (optional): File pattern to limit which files this rule applies to (e.g., `*.md`) **How it works:** - During auto-merge, if a local file exists and preserve rules are configured - Each line in the upstream content is checked against the preserve patterns - If a line matches a pattern AND the local file has a different version of that line - The local version is preserved instead of being overwritten by upstream ### Step 3: Initialize State Create `<PROJECT_ROOT>/sync-state.json` with the starting commit: ```json { "upstream_remote": "upstream", "upstream_branch": "main", "last_synced_commit": "<STARTING_SHA>", "last_sync_time": null, "sync_history": [] } ``` ### Step 4: Run Sync ```bash cd <PROJECT_ROOT> node --import tsx scripts/sync-upstream.ts --from <COMMIT_SHA> ``` | Flag | Description | |------|-------------| | `--from <sha>` | Start from this commit (overrides state file) | | `--dry-run` | Preview only, no changes | | `--no-fetch` | Skip `git fetch` (data already local) | | `--skip-build-check` | Skip `npm run docs:build` verification | ### Step 5: Review & Merge 1. Check terminal output for auto-merged, manual-review, and **unmapped** files 2. Review `sync-conflict-report.md` for details (includes suggested mapping JSON for unmapped paths) 3. If unmapped new upstream dirs appear: add suggested entries to `.upstream-mapping.json`, then re-run sync 4. Handle manual-review files (large diffs >= 30 lines, deleted files, SUMMARY.md): ```bash git show upstream/main:<upstream-path> # view upstream version # edit local file, then stage + commit ``` 5. Merge into main: ```bash git checkout main git merge sync/<timestamp> git branch -d sync/<timestamp> ``` ## Auto vs Manual Classification | Condition | Handling | |-----------|----------| | New file (added) within mapped path | Auto-merge — write new local file + `git add` | | Rename within mapped path | Auto-merge — write new path, remove old local path | | < 30 lines, no local modifications | Auto-merge | | >= 30 lines changed | Manual review — keep local, do not overwrite | | Has local-only modifications | Manual review | | File deleted upstream | Manual review — **keep local file** until reviewed | | `SUMMARY.md` (any depth) | Always manual review | | Path matches no mapping rule | **Not synced** — listed in report with suggested mapping | ## Unmapped New Paths Upstream may add new top-level directories that are not in `.upstream-mapping.json`. These are **not** silently ignored: 1. Phase 2 prints each unmapped path with kind (`added` / `modified` / …) 2. Suggests mapping entries inferred from existing local prefix (e.g. `docs/zh/`) 3. Writes them into `sync-conflict-report.md` under **Unmapped Upstream Paths** 4. After you add the mappings, re-run sync to pull those files ## Integrity Verification Post-merge checks: - Directory targets exist and contain non-empty `.md` files - File targets exist and are non-empty - New files written from upstream are listed - No files lost compared to pre-merge snapshot (deleted upstream files are not auto-removed) ## Sync Phases 1. **Prepare** — read state, fetch upstream, detect new commits 2. **Analyse** — diff files, apply path mapping + exclude, classify auto vs manual, warn on unmapped 3. **Backup** — create `backup/pre-sync-<ts>` and `sync/<ts>` branches 4. **Merge** — file-by-file via `git show <commit>:<path>`, write/add mapped local paths, stage dirs + files + applied paths 5. **Report** — update state, print summary (incl. unmapped + suggestions), save report