lakebase-scm-workflows

# lakebase-scm-workflows – agent contract Agent-facing contract: operating rules (`.env`, git hooks, credential single-seam), concrete code patterns for each substrate primitive, and reference pointers. For the human-facing overview (prerequisites, installation, prompts, journey, CLI cheat sheet) see [`README.md`](README.md). **FIRST**: load the parent `databricks-lakebase` skill for Lakebase Postgres CLI basics (project / branch / endpoint shapes, name formats, "never delete the production branch" rule). This skill composes on top of it. ## Project state – when `.env` matters The substrate API takes explicit args (`instance`, `branch`, etc.) on every public function – agents can drive every operation without a project `.env` at all. **But** when an agent is acting AS the developer in a checked-out paired project, the project's `.env` is the source of truth for "which Lakebase branch is this workspace currently paired with." | Agent context | `.env` contract | |---|---| | In a checked-out paired project (Claude Code / Cursor / Genie Code on a developer's machine) | **Respect it.** Read `LAKEBASE_PROJECT_ID` to derive `instance`. After `git checkout`, call `syncEnvToCurrentBranch({ cwd })` so `.env` matches the new branch – otherwise the bundled CI scripts (`refresh-token.sh`, `flyway-migrate.sh`) and the git hooks operate on stale credentials. | | Sandbox / no workspace (Claude Desktop, OpenAI Agent Builder, exploratory) | **Ignore it.** Pass `instance` and `branch` explicitly per call. Substrate never requires `.env`. | | Bootstrapping a new project | **N/A.** `createProject` creates the `.env` for you as step 7. No `.env` exists before that. | **Connection-block keys** (managed by `syncEnvToCurrentBranch` / `updateEnvConnection` / `post-checkout.sh`): ``` LAKEBASE_PROJECT_ID=proj-abc LAKEBASE_BRANCH_ID=feature-x LAKEBASE_HOST=ep-....database.cloud.databricks.com LAKEBASE_ENDPOINT=primary DB_USERNAME=user@databricks.com ``` This is connection METADATA only , **no DB token is persisted**. The app + its migrations (alembic / knex / flyway) mint a fresh short-lived Lakebase credential at runtime from this metadata (via the `lakebase-get-connection` seam), so a token can never go stale on disk. Set `DATABASE_URL` explicitly to override with a fixed credential (CI secret / Docker). This block is the rewritten set on every branch switch; anything else in `.env` is preserved verbatim. **Project-level keys** (written once by `writeEnvFile` during project bootstrap, never rewritten): ``` DATABRICKS_HOST=https://workspace.cloud.databricks.com LAKEBASE_PROJECT_ID=my-app ``` If you're an agent dropping into a project mid-session, read these first to know what `instance` to pass to every subsequent operation. ## Workflow state surface – `.lakebase/workflow-state.json` The SCM workflow is a five-state machine: `scaffold-complete` → `feature-claimed` → `pr-ready` → `ci-green` → `merged`. Its gate surface is a single JSON file at the project root, validated against [`scm-workflow-state.schema.json`](../../scripts/lakebase/scm-workflow-state.schema.json). The current state plus its invariants (feature id, branch, parent branch, Lakebase UID, PR URL, CI URL) are persisted there. Inspect it via: ```bash lakebase-scm-state # human-readable, with gate ladder lakebase-scm-state --json --pretty # machine-readable report lakebase-scm-state --project-dir ~/repos/x # different project root ``` Read / write programmatically: ```ts import { readWorkflowState, writeWorkflowState, initWorkflowState, describeGates, } from "@databricks-solutions/consort"; const s = readWorkflowState(projectDir); // ScmWorkflowState | null writeWorkflowState(projectDir, { // validates first; atomic write ...initWorkflowState({ projectId: "demo", tierTopology: 2 }), }); const gates = describeGates(s!); // gate ladder for tooling ``` Phase A (advisory data layer): `lakebase-create-project` seeds the `scaffold-complete` row at end-of-scaffold and `lakebase-scm-state` reads it. A seed failure surfaces as a project warning, not a scaffold abort. Phase B (first blocking transition): `lakebase-scm-claim-feature-branch` is the canonical "start a new feature" verb. It enforces its precondition in code (state must be `scaffold-complete` or `merged`), calls the substrate primitive `createFeaturePairedBranch` (Lakebase branch + git branch + .env sync, 30-day TTL), and advances the state file to `feature-claimed`. /design's pre-hook invokes it; the substrate-only-path invariant is enforced through this bin. ```bash lakebase-scm-claim-feature-branch initial-domain # claim lakebase-scm-claim-feature-branch initial-domain --json # machine-readable lakebase-scm-claim-feature-branch hotfix-x --parent main # override tier-default parent lakebase-scm-claim-feature-branch initial-domain # idempotent re-run = no-op ``` Exit codes: `0` success (incl. idempotent no-op), `1` no state file, `2` precondition refused, `3` substrate failure. Programmatic equivalent: ```ts import { claimFeatureBranch } from "@databricks-solutions/consort"; const { state, paired, alreadyClaimed } = await claimFeatureBranch({ projectDir, featureId: "initial-domain", }); ``` ### Full SCM CLI surface (phase C) The workflow is driven entirely by CLI bins, one per transition. Each enforces its precondition in code, calls the underlying substrate primitives, and writes the new state row. | Bin | Transition | Substrate it wraps | |-----|------------|--------------------| | `lakebase-scm-state` | inspect (read-only) | (reads `.lakebase/workflow-state.json`) | | `lakebase-scm-doctor` | diagnose (read-only) | cross-checks state + git + Lakebase + .env | | `lakebase-scm-adopt-state` | seed for existing projects | listBranches + getBranchByName + getCurrentBranch | | `lakebase-scm-recover-orphans [--claim]` | retroactively pair orphan git branches | createFeaturePairedBranch per orphan | | `lakebase-scm-claim-feature-branch <id>` | scaffold-complete \| merged -> feature-claimed | createFeaturePairedBranch | | `lakebase-scm-abandon-feature` | feature-claimed -> scaffold-complete | deletePairedBranch + git checkout | | `lakebase-scm-prepare-pr` | feature-claimed -> pr-ready | git push + createPullRequest | | `lakebase-scm-wait-ci [--timeout-sec N]` | pr-ready -> ci-green | getPullRequest poll loop | | `lakebase-scm-merge [--method squash\|merge\|rebase]` | ci-green -> merged | mergePairedPullRequest + git cleanup | | `lakebase-reconcile-tier --branch <tier>` | reconcile a shared tier whose DB is ahead of code (destructive; refuses when not db-ahead) | reconcileTierBranch (drop named orphan tables + stamp the tier to the code head) | All bins support `--project-dir <dir>` (default cwd), `--json`, `--pretty`, and `--help`. End-to-end usage: ```bash # Scaffold a fresh project (Step 8c seeds scaffold-complete). lakebase-create-project --tiers 2 ... # Existing projects opt in once. lakebase-scm-adopt-state # Or, if they have pre-phase-C orphan branches: lakebase-scm-recover-orphans # detect-only lakebase-scm-recover-orphans --claim # pair every orphan via the substrate # Per-feature cycle. lakebase-scm-claim-feature-branch initial-domain # ... write code, run tests ... lakebase-scm-prepare-pr lakebase-scm-wait-ci lakebase-scm-merge # state is now merged; the next claim returns from merged to feature-claimed. # Inspect / diagnose at any point. lakebase-scm-state --json --pretty lakebase-scm-doctor ``` Exit-code conventions across all bins: `0` success / idempotent no-op / clean doctor report, `1` no state file (or doctor warnings only), `2` precondition refused (or doctor failures), `3` substrate failure. `lakebase-scm-wait-ci` adds `3` for CI failure (state unchanged) and `4` for timeout (state unchanged). `lakebase-scm-merge` also returns `4` when the downstream migrate fails or times out (the git + PR state IS already merged). ### Phase C: substrate-only-path is now enforced The post-checkout git hook (`templates/project/common/scripts/post-checkout.sh`) used to silently create Lakebase branches as a fallback for orphan git branches. Phase C retires that fallback: a `git checkout -b feature/foo` with no prior substrate call leaves `.env` untouched and the hook prints a clear error pointing the user at `lakebase-scm-claim-feature-branch` or `lakebase-scm-recover-orphans`. The substrate is the only path; the SCM workflow is how that path is enforced in code. `lakebase-scm-merge` waits on the downstream migrate by default: after merging it blocks until the parent-branch migrate workflow applies the migrations, so git and the Lakebase schema do not diverge. `--no-wait-migrate` opts out (returns as soon as the PR merges + local syncs), and a migrate that fails or times out exits `4` with the git + PR state already merged. For targeted remediation, `lakebase-scm-doctor --fix <finding-id>` applies the fix for one finding (supported ids come from `FIXABLE_FINDING_IDS`) and attaches a post-fix report. ## Sync without an IDE – the git hooks The construct that keeps a Lakebase branch and a git branch in sync in a plain terminal session (no extension, no explicit substrate call) is the **bundled git hooks** that `scaffoldAll` / `installHooks` drops into `.git/hooks/` during project bootstrap. They are the default-on automatic sync mechanism. Agents driving raw `git` commands inherit them for free. | Hook | Fires on | What it does | |---|---|---| | `post-checkout` | `git checkout <branch>` | Reads new current branch → finds matching Lakebase branch → mints fresh credential → rewrites `.env` connection block. **The primary sync mechanism.** | | `post-merge` | `git merge` | Runs Flyway migrations against the now-current Lakebase branch so schema catches up. | | `pre-push` | `git push` | Schema-diff guard – surfaces unmigrated changes before remote sees them. | | `prepare-commit-msg` | `git commit` | Embeds Lakebase branch context in commit messages so the schema-diff CI workflow can find them. | **Practical implications for an agent:** 1. **Don't fight the hooks.** If you run `git checkout feature-x` in a paired project, `.env` auto-updates. Don't also call `syncEnvToCurrentBranch` defensively – let the hook own that side of the workflow. 2. **Hooks don't create branches.** They sync state after-the-fact. To CREATE a Lakebase branch (which has no git equivalent), use the substrate's `createPairedBranch` – it creates the Lakebase side first, then `git checkout -b` triggers the hook to populate credentials. 3. **If hooks aren't installed, re-arm them.** Some workflows clone a paired project without scaffolding (e.g. cloning someone else's checkout). The substrate's `installHooks(projectDir)` is the recovery – copies `scripts/post-checkout.sh` and siblings into `.git/hooks/` with the right permissions. 4. **For pure-API sessions (no checkout) the hooks are irrelevant.** A Claude Desktop sandbox or OpenAI Agent Builder session that just calls `getConnection({ instance, branch })` doesn't have a `.git/` to install hooks into – and doesn't need them. The hooks only matter when an agent (or human) is driving a working tree with `git` commands. The bundled hook scripts live in `templates/project/common/scripts/`. ## Credential handoff – two helpers, one pattern Two narrow auth seams – one for Lakebase, one for GitHub. Both follow the same shape: single module, dynamic-runtime fallback chain, CI grep guard preventing any other file from resolving credentials directly. ### GitHub ```bash lakebase-github-token # print token to stdout lakebase-github-token --diagnose # which sources are configured ``` ```ts import { resolveGitHubToken } from "@databricks-solutions/consort"; const token = await resolveGitHubToken(); ``` Fallback: `GITHUB_TOKEN` env → VS Code `getSession` (extension host only, via dynamic `import('vscode')`) → `gh auth token` → clear error. Scopes: `['repo', 'workflow', 'delete_repo']`. Full docs: [`references/github-auth.md`](references/github-auth.md). ### Lakebase Every workflow op that touches Lakebase resolves credentials through a single seam: ```bash lakebase-get-connection --output dsn --instance <id> --branch <name> # -> libpq URL string (use for Flyway, Alembic, psql) ``` ```ts import { getConnection } from "@databricks-solutions/consort"; const pool = await getConnection({ output: "pool", instance, branch }); // -> @databricks/lakebase pg.Pool with refresh-on-connect ``` DSN and Pool resolve to the same database via the same OAuth substrate. Never call `databricks postgres generate-database-credential` from anywhere else in your code – a CI grep guard fails the build if you do. Full docs: [`references/get-connection.md`](references/get-connection.md). ## Operations Concrete invocations per primitive, in user-journey order. The agent reads these to know what to call; humans see the conversational equivalent in [`README.md`'s "How to use"](README.md#how-to-use). ### 1. Create-project End-to-end project bootstrap. ```bash lakebase-create-project \ --project-name proj-checkout \ --parent-dir ~/code \ --databricks-host https://workspace.cloud.databricks.com \ --github-owner my-org \ --language java \ --runner self-hosted # -> JSON on stdout: { projectDir, githubRepoUrl, lakebaseProjectId, lakebaseDefaultBranch, warnings } # Local-only (no GitHub side effects): lakebase-create-project --project-name proj-checkout --parent-dir ~/code \ --databricks-host https://workspace.cloud.databricks.com --no-github # Wire Playwright into the project so [E2E]-tagged AC rows have a runner. # Default-on for --language nodejs; opt in elsewhere with --enable-e2e or # turn off with --no-e2e: lakebase-create-project ... --language nodejs --enable-e2e # Skip the .claude/commands/{design,build}.md scaffold (for projects # that already have their own slash commands or non-Claude-Code consumers): lakebase-create-project ... --skip-commands ``` ```ts import { createProject } from "@databricks-solutions/consort"; const result = await createProject({ projectName: "proj-checkout", parentDir: process.env.HOME + "/code", databricksHost: "https://workspace.cloud.databricks.com", githubOwner: "my-org", language: "java", runnerType: "self-hosted", enableTdd: true, // default: true – lays down .sftdd/ scaffold enableE2e: undefined, // default: true for nodejs, false otherwise. // Explicit boolean overrides the language default. skipCommands: false, // default: false – writes .claude/commands/{design,build}.md }); ``` Eleven-step orchestration. Non-fatal failures (CI secrets sync, runner setup, hook/workflow verification) land in `result.warnings[]`. Hard-fatal errors (input validation, GitHub repo creation, Lakebase project creation, git operations, push rejection on workflow scope) throw. ### 2. Branch lifecycle Lakebase branch CRUD plus paired git+Lakebase+.env ops. The `lakebase-branch` bin is the shell-friendly entrypoint; same operations import directly from the package for JS/TS hosts. ```bash # Lakebase-only branch lifecycle lakebase-branch list --instance proj-checkout lakebase-branch show --instance proj-checkout --branch feature-add-orders lakebase-branch create --instance proj-checkout --branch feature-add-orders --parent staging lakebase-branch delete --instance proj-checkout --branch feature-add-orders