merge-upstream

name	merge-upstream
description	Merge new upstream releases of nushell and reedline into Shannon

Merge Upstream

Shannon tracks nushell and reedline as git subtrees under nushell/ and reedline/. Every few nushell releases, we pull upstream to pick up new features and fixes. This skill captures what actually works — because scripts/sync-upstream.sh alone is not enough when upstream has drifted.

Why the script alone doesn't work

scripts/sync-upstream.sh runs git subtree pull for nushell and reedline, then cargo build && cargo test. That's fine for tiny drifts. For real upstream releases it fails because:

1. Subtree auto-merge is noisy. A hundred or more files will conflict — most of them files Shannon never touched. The auto-merger gets confused by rename chains and workspace-wide edits.
2. Auto-merged files end up half-upstream, half-fork. Even files that don't show conflicts can end up with stale content that references removed APIs. You won't see this until cargo build fails deep in nu-protocol or nu-parser.
3. New upstream files get missed. When auto-merge fights with the subtree prefix, new files upstream added may not appear in our tree at all — the build will complain about missing modules.
4. Shannon's own src/ lags upstream API. src/main.rs and src/run.rs are copied from nushell's binary and drift every release.
5. Root Cargo.toml version pins don't auto-bump. The Shannon crate still references the old 0.N.0 nu-* versions.
6. Reedline must move in lockstep. Nushell's workspace pins a specific reedline version; if you only pull nushell, the build fails because our vendored reedline is stale.

Shannon's fork surface (the only 10 files we preserve)

Keep these exact files across upgrades. Everything else in nushell/ should come from upstream verbatim:

• nushell/Cargo.toml — workspace reedline path dep, shannon package renames (shannon-nu-cli, shannon-nu-lsp), shannon crate versions
• nushell/crates/nu-cli/Cargo.toml — name = "shannon-nu-cli", tree-sitter deps for BashHighlighter
• nushell/crates/nu-cli/src/bash_highlight.rs — NEW, tree-sitter-based bash syntax highlighter
• nushell/crates/nu-cli/src/mode_dispatcher.rs — NEW, ModeDispatcher trait
• nushell/crates/nu-cli/src/lib.rs — declares mod bash_highlight, mod mode_dispatcher, and re-exports BashHighlighter, ModeDispatcher, ModeResult
• nushell/crates/nu-cli/src/repl.rs — dispatch hook in loop_iteration() that forwards to ModeDispatcher::execute() when $env.SHANNON_MODE is not "nu"; also a few smaller tweaks
• nushell/crates/nu-cli/src/nu_highlight.rs — small tweak
• nushell/crates/nu-command/src/platform/input/input_.rs — small tweak
• nushell/crates/nu-lsp/Cargo.toml — name = "shannon-nu-lsp", references shannon-nu-cli

Verify this list against the current state before you start:

git diff --stat <last-nushell-import-commit>..HEAD -- nushell/

If new Shannon-modified files appear, add them to the preserve list below.

The reliable procedure

Work on a branch — never on main.

1. Preflight

Clean working tree. Fetch upstream. Count the drift.

git status                                       # must be clean
git fetch upstream-nushell upstream-reedline
git log --oneline <last-merge-base>..upstream-nushell/main | wc -l

Large drifts (100+ commits) are the norm — that's fine, just plan for conflicts.

2. Branch off

git checkout -b upgrade/nushell-$(date +%Y-%m-%d)

3. Pull nushell (expect conflicts)

git subtree pull --prefix nushell upstream-nushell main \
  -m "Merge nushell upstream $(date +%Y-%m-%d)"

This will fail with "Automatic merge failed". That is expected.

4. Save Shannon's fork files to /tmp

Before doing anything destructive:

mkdir -p /tmp/shannon_patches
cp nushell/Cargo.toml                                       /tmp/shannon_patches/Cargo.toml
cp nushell/crates/nu-cli/Cargo.toml                         /tmp/shannon_patches/nu-cli-Cargo.toml
cp nushell/crates/nu-cli/src/bash_highlight.rs              /tmp/shannon_patches/bash_highlight.rs
cp nushell/crates/nu-cli/src/mode_dispatcher.rs             /tmp/shannon_patches/mode_dispatcher.rs
cp nushell/crates/nu-cli/src/lib.rs                         /tmp/shannon_patches/lib.rs
cp nushell/crates/nu-cli/src/nu_highlight.rs                /tmp/shannon_patches/nu_highlight.rs
cp nushell/crates/nu-cli/src/repl.rs                        /tmp/shannon_patches/repl.rs
cp nushell/crates/nu-command/src/platform/input/input_.rs   /tmp/shannon_patches/input_.rs
cp nushell/crates/nu-lsp/Cargo.toml                         /tmp/shannon_patches/nu-lsp-Cargo.toml

The unresolved conflict markers in those files are fine — they're snapshots, not for reuse. What you actually need from them is the Shannon side of each hunk, which you'll recreate by hand in step 7. In practice the easier workflow is: commit the broken merge first (step 5), then re-export clean Shannon versions from the main branch:

git show main:nushell/Cargo.toml > /tmp/shannon_patches/Cargo.toml
# ... and so on

5. Commit the busted merge so you have a clean slate

Don't try to hand-resolve 100+ conflicts. Just stage whatever's there and commit it — you're about to overwrite the tree anyway.

git checkout --theirs -- $(git diff --name-only --diff-filter=U)
git add -A
git -c core.editor=true commit --no-edit

6. Wholesale-replace the nushell/ tree with upstream

This is the step that makes everything else tractable. Wipe nushell/ and re-populate from upstream-nushell/main:

git rm -rqf nushell/
mkdir -p nushell
git archive upstream-nushell/main | tar -x -C nushell/

You now have a pristine copy of upstream's tree at nushell/, free of any auto-merge weirdness.

7. Re-apply Shannon's fork files

cp /tmp/shannon_patches/Cargo.toml          nushell/Cargo.toml
cp /tmp/shannon_patches/nu-cli-Cargo.toml   nushell/crates/nu-cli/Cargo.toml
cp /tmp/shannon_patches/bash_highlight.rs   nushell/crates/nu-cli/src/bash_highlight.rs
cp /tmp/shannon_patches/mode_dispatcher.rs  nushell/crates/nu-cli/src/mode_dispatcher.rs
cp /tmp/shannon_patches/lib.rs              nushell/crates/nu-cli/src/lib.rs
cp /tmp/shannon_patches/nu_highlight.rs     nushell/crates/nu-cli/src/nu_highlight.rs
cp /tmp/shannon_patches/repl.rs             nushell/crates/nu-cli/src/repl.rs
cp /tmp/shannon_patches/input_.rs           nushell/crates/nu-command/src/platform/input/input_.rs
cp /tmp/shannon_patches/nu-lsp-Cargo.toml   nushell/crates/nu-lsp/Cargo.toml

Then update the Shannon files for upstream API churn:

• nushell/Cargo.toml — bump all version = "0.OLD.0" entries in the [dependencies] block to match the new upstream version. Keep the shannon-nu-cli / shannon-nu-lsp package renames and Shannon crate versions. Bump reedline in [workspace.dependencies] to the new version and keep path = "../reedline". The [workspace.package] and [[test]] blocks may be new from upstream — preserve them.
• nushell/crates/nu-cli/Cargo.toml — bump all version = "0.OLD.0" in both [dev-dependencies] and [dependencies]. Add rust-version.workspace = true and autotests = false if upstream introduced them.
• nushell/crates/nu-lsp/Cargo.toml — same pattern.

8. Pull reedline

git subtree pull --prefix reedline upstream-reedline main \
  -m "Merge reedline upstream $(date +%Y-%m-%d)"
# Resolve Cargo.lock conflict by taking upstream:
git checkout --theirs -- reedline/Cargo.lock
git add reedline/Cargo.lock
git -c core.editor=true commit --no-edit

Reedline has no Shannon-side changes, so conflicts are minimal (usually just Cargo.lock).

9. Regenerate Cargo.lock files

rm nushell/Cargo.lock
(cd nushell && cargo generate-lockfile)

The root Cargo.lock regenerates on the next cargo build.

10. Bump root Cargo.toml

In /Users/ryan/dev/shannon/Cargo.toml, update:

• Every nu-* = { version = "0.OLD.0", ... } to the new version
• reedline = { version = "0.OLD.0", ... } to the new version

A sed one-liner works if the old version is unique:

sed -i '' 's/version = "0.111.0"/version = "0.112.2"/g' Cargo.toml
sed -i '' 's/version = "0.46.0"/version = "0.47.0"/g' Cargo.toml

(macOS sed uses -i ''. Linux: sed -i.)

11. Update Shannon's src/main.rs and src/run.rs for API churn

Shannon's src/ is copied from nushell's binary and drifts every release. Diff against upstream to find what changed:

diff src/main.rs nushell/src/main.rs
diff src/run.rs  nushell/src/run.rs

Common changes:

• std::time::Instant → nu_utils::time::Instant. Nushell migrated to its own Instant wrapper. Replace everywhere in Shannon's src/.
• nu_protocol::location!() removed. Calls to IoError::new_internal_with_path(err, msg, location!(), path) now take only (err, msg, path) — drop the location!() argument.
• ShellError::GenericError → ShellError::Generic. (Currently emits deprecation warnings; not a build failure yet.)
• evaluate_repl signature changes. Check the argument list against upstream if you get a type mismatch.

The diff against upstream's equivalent file is the fastest way to find all call sites that need updating.

12. Build

cargo build

First build errors will usually be in nu-parser or nu-protocol complaining about missing exports. If you see this after a wholesale tree replace, it's almost always stale incremental compilation artifacts from an earlier failed build. Force a rebuild of the affected crate:

touch nushell/crates/nu-experimental/src/lib.rs  # or whichever crate is stuck
cargo build

Avoid cargo clean — per nushell/AGENTS.md, it just wastes compile time.

Once nushell/ compiles, the next errors will be in shannonshell itself (src/main.rs, src/run.rs) — those are the API-churn fixes from step 11.

13. Smoke test

./target/debug/shannon --version
./target/debug/shannon

In the interactive shell:

1. Type a nushell command (e.g. ls) — verify nu mode works
2. Press Shift+Tab — verify mode switches to bash
3. Type a bash command (e.g. echo $HOME) — verify bash mode works
4. Press Shift+Tab — verify it switches back to nu
5. Verify env vars propagate across the switch (e.g. cd /tmp in bash, then back to nu and check pwd)

The build passing is not sufficient — Shannon's ModeDispatcher hook lives in repl.rs, which upstream rewrites frequently. A merge can compile fine but silently break the dispatcher.

14. Commit the work

At this point you should have on the branch:

1. The busted-merge commit (nushell subtree pull)
2. The reedline merge commit
3. One or two commits for the wholesale tree replace + Shannon fork re-application + root Cargo.toml / src/ API updates

Merge to main when ready:

git checkout main
git merge --no-ff upgrade/nushell-$(date +%Y-%m-%d)

Things that will bite you

• Don't use --squash with git subtree. Shannon's AGENTS.md explicitly forbids it. Full history across merged projects must be preserved for blame/log/bisect.
• Don't hand-resolve 100+ conflicts. Wholesale replace is faster and correct. Conflict-by-conflict resolution leaves stale auto-merged content in files you don't notice until build time.
• Don't forget reedline. Pull both or the build will fail on version pinning.
• Don't forget the root Cargo.toml. Bumping only nushell/Cargo.toml is not enough.
• Don't skip the interactive smoke test. cargo build does not exercise the dispatcher hook.
• The scripts/sync-upstream.sh script is not the source of truth. This skill is. Update the script if you want, but don't rely on it alone for real upgrades.

After the upgrade

Consider opening an issue under issues/ to track any cleanup work — deprecation warnings to address, features upstream added that Shannon could expose (e.g. ExternalHinter in repl.rs), or new commands that should be wired up.