| name | upstream-sync |
| description | Safely pulls upstream acestep.cpp changes into the HOT-Step engine fork without destroying its integration hooks. Use when syncing/merging upstream acestep.cpp updates, verifying or repairing hook files, or diagnosing "unresolved external symbol hotstep_sampler_linked_" or dead solvers/schedulers/guidance after a sync. |
Syncing with upstream acestep.cpp
When to use this skill
- Pulling new commits from upstream
acestep.cpp (https://github.com/ServeurpersoCom/acestep.cpp) into this fork's engine/.
- Checking whether a sync is pending, or repairing a botched one.
- Diagnosing: linker error
unresolved external symbol hotstep_sampler_linked_, or a build that compiles clean but all solvers/schedulers/guidance modes stopped working.
Background you need (no prior context assumed): The C++ engine in engine/ is a fork of acestep.cpp, a GGML port of the ACE-Step music model. The generation pipeline is LM → DiT → VAE, where DiT = Diffusion Transformer, the denoising model. HOT-Step replaces upstream's DiT sampling loop (dit-sampler.h) with its own (hot-step-sampler.h) that adds 17 solvers, 9 schedulers, guidance modes, and Lua plugins. The fork is not a git merge — it is file-copy based, using three zones:
| Zone | What | Sync strategy |
|---|
| Upstream Zone | Most of engine/src/*.h / *.cpp — kept byte-identical to upstream | Direct Copy-Item from a vanilla clone |
| HOT-Step Zone | engine/src/hot-step-*, engine/src/solvers/, engine/src/schedulers/, engine/src/guidance/, engine/src/mastering.h, engine/src/adapter-runtime.h, engine/src/adapter-merge.h, engine/src/model-registry.h, engine/tools/hot-step-server.cpp — never exists upstream | Never conflicts; never copy |
| Modified Upstream Zone | Exactly 3 files: pipeline-synth-ops.cpp, model-store.h, dit.h — upstream files carrying HOT-Step #include hooks | Manual merge only, NEVER direct-copy |
⚠ Zone model drift (verified 2026-07-14): two of this table's assumptions no longer hold.
src/solvers/ now EXISTS upstream (added 2026-05-06, commit 8974778 — modular Euler/SDE/DPM++3M/STORK4 dispatch). Its 7 filenames collide with the fork's own engine/src/solvers/ (created independently 2026-04-18, commit 889c250, different interface). The Phase 2 copy loop MUST also exclude everything under src/solvers/ — a direct copy clobbers HOT-Step's native solvers with incompatible upstream versions.- The Upstream Zone has eroded. As of 2026-07-14, 11 of the 13 upstream-touched files since
31cc9ea are fork-modified (e.g. qwen3-lm.h carries safetensors loading via fork-only weight-source.h/config-json.h; backend.h, dit-graph.h, vae.h, request.*, wav.h, qwen3-enc.h, pipeline-synth.* also differ). Do NOT assume direct-copy safety from this table — before copying any file, verify pristineness: diff the fork's copy against git show <lastSynced>:src/<file> from the upstream clone (with --strip-trailing-cr). Pristine → copy; modified → manual merge.
engine/tools/ace-server.cpp is upstream's server kept as a direct-copy reference only — not compiled. The shipped ace-server binary is built from engine/tools/hot-step-server.cpp (see engine/CMakeLists.txt:416-418).
The full workflow doc lives at docs/plans/upstream-sync-workflow.md, which is gitignored (local-only) — other machines will not have it. This skill is the portable distillation, with two known errors in that doc corrected (see reference.md).
Golden rules (hard constraints)
- NEVER direct-copy the 3 Modified Upstream Zone files (
pipeline-synth-ops.cpp, model-store.h, dit.h). WHY: copying pipeline-synth-ops.cpp wholesale is the silent failure — it compiles clean but kills every HOT-Step sampler feature (see hook table below). This actually happened once; the whole defense system exists because of it.
- ALWAYS run
engine/verify-hooks.ps1 after copying files and BEFORE building. WHY: it catches hook loss in seconds; a build wastes minutes and a silent regression wastes hours.
- Commit the fork locally before starting. WHY: file-copy sync has no git-merge undo; the commit is your only rollback point.
- NEVER push the
v1.5-upstream-sync-* tag (or anything without explicit user approval). WHY: any pushed v* tag triggers the full multi-platform CI release build. The tag is fine locally.
- Rebuild via
dev-rebuild.bat at repo root, NEVER engine/build.cmd directly. WHY: the app may be running without you knowing (you cannot reliably tell); the Node server auto-respawns ace-server on crash → infinite respawn + file-lock loop. dev-rebuild.bat handles clean shutdown and is a safe no-op when nothing is running.
- NEVER
cmake --build . --clean-first. WHY: CUDA kernel recompile is 20+ min. Stale .obj? Delete only engine/build/acestep-core.dir/ and engine/build/Release/acestep-core.lib.
- Never add HOT-Step code to Upstream Zone files (new behavior goes in a
hot-step-* file), and never add hot-step-* files to the upstream clone. WHY: keeps future syncs a trivial copy.
- Git hygiene: all work on
master, stage explicit paths only, never git add -A.
The three hook files (verified line numbers)
| File | Hook | If clobbered by direct copy |
|---|
engine/src/pipeline-synth-ops.cpp | #include "hot-step-sampler.h" at line 9 (replaces upstream's dit-sampler.h). Lines 10–12 are also fork-only: hot-step-sampler-trt.h, adapter-trt.h, stream-pipeline.h | SILENT at compile time — all solvers/schedulers/guidance/custom timesteps/DCW/APG go dead. Caught only by the linker sentinel |
engine/src/model-store.h | #include "hot-step-params.h" at line 53 (provides AdapterGroupScales) | Compile error (self-catching) |
engine/src/dit.h | #include "adapter-merge.h" (line 11) + #include "adapter-runtime.h" (line 13); line 12 hot-step-build-flags.h is also fork-only | Compile error (self-catching) |
Why the sampler hook failure is silent: the hook is a substitution, not an addition. Upstream dit-sampler.h and fork hot-step-sampler.h both provide a function named dit_ggml_generate. A whole-file copy of upstream pipeline-synth-ops.cpp replaces both the include AND the call site (pipeline-synth-ops.cpp:1503) with mutually-consistent upstream versions — everything compiles, HOT-Step's sampler simply stops being called.
The defense — linker sentinel: engine/src/hot-step-sampler.h:1312 defines external-linkage symbol hotstep_sampler_linked_; engine/tools/hot-step-server.cpp:51-52 references it. If the include reverts to dit-sampler.h, the symbol vanishes and the link fails with unresolved external symbol hotstep_sampler_linked_. That exact error means: re-hook hot-step-sampler.h in pipeline-synth-ops.cpp. Mechanism detail: reference.md.
verify-hooks.ps1
powershell -File "D:\Ace-Step-Latest\hot-step-cpp\engine\verify-hooks.ps1"
Exit 0 = intact, exit 1 = broken. It runs 5 checks (not 3 — the script is authoritative over the docs): the three hook files above, plus hot-step-server.cpp → hot-step-params.h include, plus the sentinel string in hot-step-sampler.h (engine/verify-hooks.ps1:15-68).
The sync procedure
Prerequisite: fork committed locally (Rule 3). The vanilla upstream clone lives at D:\Ace-Step-Latest\acestepcpp\acestep.cpp (a separate repo; if this machine lacks it, git clone https://github.com/ServeurpersoCom/acestep.cpp there first).
Phase 1 — Fetch and assess
git -C D:\Ace-Step-Latest\acestepcpp\acestep.cpp pull origin master
Read the last-synced commit from the marker. The marker uses KEY=VALUE lines — parse the UPSTREAM_COMMIT= line (the workflow doc's "first non-comment line" snippet is WRONG for this format):
$lastSynced = ((Get-Content "D:\Ace-Step-Latest\hot-step-cpp\engine\UPSTREAM_SYNC" | Where-Object { $_ -match '^UPSTREAM_COMMIT=' }) -replace '^UPSTREAM_COMMIT=','').Trim()
git -C D:\Ace-Step-Latest\acestepcpp\acestep.cpp log --oneline "$lastSynced..HEAD" -- src/ tools/ace-server.cpp
git -C D:\Ace-Step-Latest\acestepcpp\acestep.cpp diff --stat "$lastSynced..HEAD" -- src/ tools/ace-server.cpp
Zero commits → already up to date, stop.
Phase 2 — Direct-copy Upstream Zone files (with hard exclusions)
cd D:\Ace-Step-Latest\acestepcpp\acestep.cpp
$excludeFiles = @("pipeline-synth-ops.cpp", "model-store.h", "dit.h") # NEVER direct-copy
$changedFiles = git diff --name-only "$lastSynced..HEAD" -- src/
foreach ($f in $changedFiles) {
$basename = Split-Path $f -Leaf
if ($excludeFiles -contains $basename) {
Write-Host "SKIP (modified upstream): $f — manual merge in Phase 3" -ForegroundColor Yellow
continue
}
Copy-Item "D:\Ace-Step-Latest\acestepcpp\acestep.cpp\$f" "D:\Ace-Step-Latest\hot-step-cpp\engine\$f" -Force
Write-Output "Copied: $f"
}
# New upstream files — any new .cpp needs an engine/CMakeLists.txt entry:
git diff --diff-filter=A --name-only "$lastSynced..HEAD" -- src/
Do NOT copy upstream src/dit-sampler.h expecting it to matter — the fork's engine/src/dit-sampler.h is dead code (nothing compiled includes it; the active sampler is hot-step-sampler.h). Copying it is harmless but see Phase 4.
Phase 3 — Manual-merge the Modified Upstream Zone
cd D:\Ace-Step-Latest\acestepcpp\acestep.cpp
foreach ($f in @("src/pipeline-synth-ops.cpp", "src/model-store.h", "src/dit.h")) {
if (git log --oneline "$lastSynced..HEAD" -- $f) { Write-Host "NEEDS MERGE: $f" -ForegroundColor Yellow }
}
For each flagged file: review git diff $lastSynced..HEAD -- src/<file> in the upstream clone, then hand-port those changes into the fork's copy without removing the hooks:
pipeline-synth-ops.cpp: keep the fork-only includes (lines 9–12) and the fork's extended dit_ggml_generate call site (line 1503). This file has diverged well beyond the include line — treat the fork's copy as the base and port upstream's diff into it, never the reverse.dit.h: keep the includes at lines 11–13 (adapter-merge.h, hot-step-build-flags.h, adapter-runtime.h).model-store.h: keep the hot-step-params.h include (line 53).
Phase 4 — Port fixes from upstream dit-sampler.h
git -C D:\Ace-Step-Latest\acestepcpp\acestep.cpp log --oneline "$lastSynced..HEAD" -- src/dit-sampler.h
If changed: review the diff and hand-port relevant bug fixes into engine/src/hot-step-sampler.h. Do NOT overwrite hot-step-sampler.h with upstream's file. If upstream changed the dit_ggml_generate signature, update both the fork's signature in hot-step-sampler.h (a superset with extra params) and the call site in pipeline-synth-ops.cpp:1503 to stay compatible.
Phase 5 — Verify hooks, build, validate
powershell -File "D:\Ace-Step-Latest\hot-step-cpp\engine\verify-hooks.ps1" # MUST pass BEFORE building
cmd /c "D:\Ace-Step-Latest\hot-step-cpp\dev-rebuild.bat" # never engine\build.cmd directly
Post-sync validation checklist:
Phase 6 — Record marker, commit, tag (do NOT push)
Write the marker in its real KEY=VALUE format (the workflow doc's Phase 6 heredoc writes an obsolete bare-hash format — do not use it):
$newHead = git -C D:\Ace-Step-Latest\acestepcpp\acestep.cpp rev-parse HEAD
@"
UPSTREAM_REPO=acestep.cpp
UPSTREAM_COMMIT=$newHead
SYNC_DATE=$(Get-Date -Format "yyyy-MM-dd")
SYNC_NOTES=<one-line summary of what came in>
"@ | Set-Content "D:\Ace-Step-Latest\hot-step-cpp\engine\UPSTREAM_SYNC"
cd D:\Ace-Step-Latest\hot-step-cpp
$shortHash = $newHead.Substring(0, 7)
git add engine/UPSTREAM_SYNC engine/src/
git commit -m "sync: upstream acestep.cpp at $shortHash"
git tag -a "v1.5-upstream-sync-$shortHash" -m "sync: upstream acestep.cpp at $shortHash"
Do not push the tag (Golden rule 4). If also updating engine/CMakeLists.txt or engine/tools/, stage those paths explicitly too.
Key files
| Path | Role |
|---|
engine/UPSTREAM_SYNC | Sync marker: last-synced upstream commit, KEY=VALUE format |
engine/verify-hooks.ps1 | 5-check hook verifier; run after copy, before build |
engine/src/pipeline-synth-ops.cpp | Modified-upstream: sampler hook (line 9) + extended call site (line 1503) |
engine/src/model-store.h | Modified-upstream: hot-step-params.h hook (line 53) |
engine/src/dit.h | Modified-upstream: adapter hooks (lines 11–13) |
engine/src/hot-step-sampler.h | The fork's active DiT sampler; linker sentinel at lines 1311–1315 |
engine/src/dit-sampler.h | Upstream's sampler — dead code in the fork, nothing includes it |
engine/tools/hot-step-server.cpp | Compiled server binary source; sentinel consumer at lines 51–52 |
engine/tools/ace-server.cpp | Upstream server, direct-copy reference only, NOT compiled |
engine/src/_backup_pre_sync/ | Git-tracked snapshot of engine/src/ taken before the 2026-04-26 sync — read-only rollback reference |
D:\Ace-Step-Latest\acestepcpp\acestep.cpp\ | Vanilla upstream clone (separate repo, machine-specific path) |
docs/plans/upstream-sync-workflow.md | Original workflow doc — gitignored, local-only, has 2 stale snippets |
Failure signatures
| Symptom | Cause | Fix |
|---|
Linker: unresolved external symbol hotstep_sampler_linked_ | pipeline-synth-ops.cpp direct-copied from upstream; include reverted to dit-sampler.h | Restore line 9 #include "hot-step-sampler.h" and re-merge the fork's call site (line 1503) plus fork-only includes (lines 10–12); re-run verify-hooks |
Compiles clean but solver/scheduler/guidance selections dead or ignored; engine log missing [DiT] Guidance: | Sampler hook clobbered AND sentinel lost (e.g. hot-step-sampler.h itself damaged) — the fully silent regression | verify-hooks checks 1 and 5; restore from git history; rebuild |
Compile error: AdapterGroupScales undefined in model-store.h | model-store.h direct-copied; lost line-53 include | Re-add #include "hot-step-params.h"; re-merge upstream diff |
Compile errors for adapter merge/runtime symbols via dit.h | dit.h direct-copied; lost lines 11–13 includes | Re-add includes; re-merge upstream diff |
| Infinite ace-server respawn + file locks during rebuild | Built via engine/build.cmd while the app runs | Use dev-rebuild.bat from repo root |
| CI release build unexpectedly kicked off | Pushed a v1.5-upstream-sync-* (or any v*) tag | Never push sync tags; delete the remote tag immediately if it happens |
$lastSynced resolves to UPSTREAM_REPO=acestep.cpp and git errors on the range | Used the workflow doc's stale marker-parse snippet | Parse the UPSTREAM_COMMIT= line (Phase 1 above) |
Institutional knowledge
- VALIDATED (it happened): a whole-file direct copy of upstream
pipeline-synth-ops.cpp once silently disabled every HOT-Step sampler feature with a clean build. The linker sentinel + verify-hooks.ps1 were built in response (design doc: docs/plans/2026-05-05-harden-upstream-sync-implementation.md, local-only).
- VALIDATED: last completed sync is upstream commit
9d38f00 on 2026-07-14 (fork commit a178b57, local tag v1.5-upstream-sync-9d38f00), done via per-file 3-way merge (git merge-file with base = previous sync point) rather than direct copy — required by the zone-model drift above. Previous sync: 31cc9ea on 2026-04-26.
- VALIDATED (2026-07-14 assessment): upstream HEAD
9d38f00; ~60 commits pending. Headline items: LM perf overhaul (qwen3-lm.h +552 lines — persistent graph arenas, static batched decode graph replay, set_rows KV writes; new headers graph-arena.h/static-graph.h, no CMake entry needed), snake autofuse pass + fused Vulkan/CUDA snake, K/V→F16 cast before flash_attn_ext (touches hook file dit.h!), WAV PCM24 fixes, GGML_CUDA_GRAPHS default ON, SYCL/Intel Arc cmake support. LM/snake work requires bumping engine/ggml (fork at e705c5fe 2026-05-25; upstream submodule at b677b63c 2026-07-08). The turbo-CFG-clobber fix (b8ba253) is ALREADY in the fork. Upstream's dit-sampler.h refactor + src/solvers/ are dead code for the fork (see zone-drift warning) — harvest math fixes only.
- VALIDATED: docs saying "3 hook files" undercount the script —
verify-hooks.ps1 runs 5 checks across 4 files. The script wins.
- VALIDATED: the workflow doc's Phase 1 marker-parse and Phase 6 marker-write snippets are stale versus the real
KEY=VALUE marker format. Use this skill's corrected versions.
- UNVALIDATED / HYPOTHESIS: swapping only the include line (not the whole file) back to
dit-sampler.h would probably be a compile error rather than silent, because the fork's call site passes an extended argument list upstream's signature lacks. Only the whole-file copy is believed truly silent. Not empirically tested.
- Rejected hardening ideas (explored, deliberately NOT implemented — don't "restore" them):
hot-step-hooks-verify.h compile guard, static_assert in the clobbered file, making dit-sampler.h a 2-line shim. Details in reference.md.
Deeper reading
- reference.md (this folder) — sentinel mechanism with code,
_backup_pre_sync/ details, DONE/NOT-DONE hardening ledger, marker file contents.
docs/plans/upstream-sync-workflow.md — original workflow (gitignored/local-only; may be absent on other machines; two snippets stale).docs/plans/2026-05-05-harden-upstream-sync-implementation.md — hardening design history (gitignored/local-only).CLAUDE.md § "Upstream sync" — the short committed summary.engine/docs/ARCHITECTURE.md — engine internals (committed).
