tinybird-deploy

# Tinybird Deployment Skill Deploy observability pipes and datasources to Tinybird Cloud. ## Requirements - **Preflight:** `which -a tb && tb --version`. If `tb` is missing or resolves to a pyenv/pip shim instead of `~/.local/bin/tb`, PATH is wrong or the Forward CLI isn't installed — fix PATH or point the user at the install docs; don't curl-pipe an installer. - Use the **Tinybird Forward CLI** as `tb`. On this machine it should resolve to `~/.local/bin/tb` and support `tb --cloud deployment create --check`. - Do **not** install or update this workflow with `pip install tinybird-cli`; that can put the Classic CLI first on PATH. - If `tb --cloud` is missing, or Tinybird says this is a Forward workspace but the CLI is Classic, fix PATH so `~/.local/bin` wins. The Classic CLI is kept only as `tb-classic`. - Run commands from `enter.pollinations.ai/observability`. - Set `TB_TOKEN` explicitly to a token with `WORKSPACE:DEPLOY` for the target workspace, and pass `--host` on every deploy command. Do not rely on `.tinyb` or `tb workspace use` for workspace selection. - `SELECT 1` succeeding doesn't mean a token can read datasources (`PIPES:READ` vs `DATASOURCES:READ` vs `WORKSPACE:DEPLOY`). Probe with a SELECT on a known datasource before relying on a token for a new path. ## Workspaces Two workspaces, same region (`gcp-europe-west2`). Pipes and datasources must be kept in sync across both. | Workspace | Receives traffic from | UI | |-----------|-----------------------|----| | `pollinations_enter` | production worker only | https://cloud.tinybird.co/gcp/europe-west2/pollinations_enter | | `pollinations_enter_staging` | staging worker + dev worker + local `npm run dev` | https://cloud.tinybird.co/gcp/europe-west2/pollinations_enter_staging | Workspace routing is token-scoped: the same regional host serves both workspaces, and the token selects the workspace. The local `.tinyb` is gitignored and must not be trusted for prod/staging selection. Set `TB_TOKEN` from the macOS Keychain, not from Enter runtime SOPS files (those only hold read/ingest tokens for the Worker). Use a staging-workspace deploy token for staging and a prod-workspace deploy token for prod. Keep tokens out of logs. Operator deploy tokens live in the macOS Keychain (same pattern as the SOPS age key): ```bash # staging (pollinations_enter_staging) TB_TOKEN="$(security find-generic-password -a "$USER" -s tinybird-staging-deploy -w)" # prod (pollinations_enter) TB_TOKEN="$(security find-generic-password -a "$USER" -s tinybird-prod-deploy -w)" ``` Always sanity-check which workspace a token targets before deploying: `tb --cloud --host "$TB_HOST" info` prints `workspace_name`. ## Directory Structure ``` enter.pollinations.ai/observability/ ├── datasources/ # Data source definitions (.datasource) │ ├── generation_event_v2.datasource │ ├── polar_event.datasource │ ├── stripe_event.datasource │ └── ... └── endpoints/ # Pipe definitions (.pipe) ├── weekly_usage_stats.pipe ├── weekly_active_users.pipe ├── daily_stripe_revenue.pipe └── ... ``` --- # Commands Set the region once per shell session and pull the staging deploy token from the Keychain: ```bash TB_HOST="https://api.europe-west2.gcp.tinybird.co" TB_TOKEN="$(security find-generic-password -a "$USER" -s tinybird-staging-deploy -w)" export TB_HOST TB_TOKEN ``` ## Step 0: Deploy From a Clean origin/main A deployment makes the workspace match the local folder, so deploy only from a clean checkout of `origin/main`. A `--check` on a feature branch is fine for validating a PR. To test an unmerged PR end to end, staging may run a feature branch that is rebased on current `origin/main` and whose check table shows only that PR's changes; redeploy staging from `origin/main` after the merge. ```bash git fetch origin && test -z "$(git status --porcelain)" && test "$(git rev-parse HEAD)" = "$(git rev-parse origin/main)" ``` Then run Step 1 and read its change table: every `new` and `deleted` resource must map to a merged PR. If one does not, stop and find out why before deploying. For production, also check every `deleted` row against the code production runs, which lags `main`: `git grep -w <name> origin/production` must find no caller, or the production deploy waits until `main` is promoted. New resources can go first. ## Step 1: Validate (Dry Run) Always validate before deploying. This is safe to run against staging. ```bash tb --cloud --host "$TB_HOST" deployment create --check --no-allow-destructive-operations ``` Example output: ``` | status | name | type | path | |----------|-----------------------|----------|--------------------------------------| | modified | weekly_usage_stats | endpoint | endpoints/weekly_usage_stats.pipe | ``` ## Step 2: Deploy If validation passes, deploy to staging first. `deployment create --wait` creates a staging deployment and waits for it to be ready. It does not promote unless `--auto` is passed. ```bash tb --cloud --host "$TB_HOST" deployment create --wait --no-allow-destructive-operations ``` Never pass `--auto` or run `deployment promote` unless the user explicitly asks for promotion. Prod deploys use the same command shape after explicitly replacing `TB_TOKEN` with the `tinybird-prod-deploy` Keychain token, but only after staging validation and verification. Deploying to both workspaces is still manual until #11127 is resolved. Staging and prod are separate approvals — don't batch them into one command, since the user may want to look at staging first. But if the instruction already names both ("deploy to staging and prod"), that is both approvals; don't re-ask in between. ## Step 3: Verify Verify the staging deployment and test the deployed endpoints with the read token for the same workspace. ```bash set -o pipefail tb --staging --cloud --host "$TB_HOST" endpoint ls tb --cloud --host "$TB_HOST" deployment ls TINYBIRD_TOKEN="$(SOPS_AGE_KEY=$(security find-generic-password -a "$USER" -s sops-age-key -w 2>/dev/null || true) \ sops -d ../secrets/staging.vars.json | jq -r '.TINYBIRD_READ_TOKEN')" curl -fsS "https://api.europe-west2.gcp.tinybird.co/v0/pipes/weekly_usage_stats.json?weeks_back=12" \ -H "Authorization: Bearer $TINYBIRD_TOKEN" | jq -e ' if has("error") then error(.error) elif (.data | type) != "array" then error("Missing data array") else .data | length end' ``` For prod verification, swap `staging.vars.json` to `prod.vars.json` and do not use `--staging`. --- # Safety Features | Flag | Description | |------|-------------| | `--check` | Validates without making changes (dry run) | | `--wait` | Waits for deployment to complete | | `--no-allow-destructive-operations` | Prevents removing datasources (default) | | `--allow-destructive-operations` | Required to delete datasources | | `--auto` | Auto-promotes a ready deployment; do not use unless explicitly requested | --- # Common Tasks ## Add a New Pipe 1. Create `.pipe` file in `endpoints/` 2. Validate against staging 3. After merge, deploy to staging from `origin/main` (Step 0), verify, then prod only when requested ## Modify Existing Pipe Same as above: edit, validate staging, merge, deploy staging from `origin/main`, verify, then prod only when requested. ## View Pipe in UI - Prod: https://cloud.tinybird.co/gcp/europe-west2/pollinations_enter/pipes - Staging: https://cloud.tinybird.co/gcp/europe-west2/pollinations_enter_staging/pipes --- # Troubleshooting ## "tb: command not found" Ensure `~/.local/bin` is on PATH, then open a new shell. Do not install the Classic CLI with `pip install tinybird-cli` for this workflow. ## "This is a Tinybird Forward workspace" / Classic CLI errors ```bash which -a tb tb --version tb --help | rg "deployment" ``` `tb` should be the Forward CLI. If the first `tb` path is under `/Library/Frameworks/Python.framework/...`, PATH is wrong; use `~/.local/bin/tb` or fix PATH. ## Validation Reports Datasource or Pipe Deletion Investigate the deletion. Restore accidentally missing definitions from a staging pull in `temp/`. For an intentional staging migration/reset within the task, verify the workspace and rerun with `--allow-destructive-operations` under root `AGENTS.md`'s disposable-staging rule. Production deletions require explicit permission. ## Pipe Timeout Issues If a pipe times out with large `weeks_back`: - Use `uniq()` instead of `uniqExact()` for user counts (~10x faster) - Avoid CTE + JOIN patterns - use single-pass queries - Consider materialized views for expensive aggregations ## Materialized View Validation Issues - Forward materialized views cannot use `UNION`; split sources into separate materialized pipes that write to the same datasource. - Validate with `deployment create --check`; older local checks are not enough. - Query-time dedup can fail validation in some endpoint pipes. Prefer dedup in the materialization when possible. --- # Important Notes - **Always use `--cloud`**: Without it, CLI tries to use Tinybird Local. - **Do NOT use `tb push`**: It is deprecated for this workflow. - **Avoid `tb deploy`**: Use explicit `deployment create` commands so promotion is never accidental. - **Destructive operations**: Follow root `AGENTS.md`'s staging and production rules; the commands above default to non-destructive validation and deployment. - **Run from observability directory**: Not from repo root.