weft-cloud-install

# Putting weft on GCP A cloud install has no machine of its own. weft's parts and programs' workers are Cloud Run services that scale to zero. Triggers that keep a connection open run on holders, which weft starts only while at least one such trigger is on. Builds run on Cloud Build, timers on Cloud Tasks, and infrastructure on Compute Engine machines. Everything weft keeps lives in a Postgres database the user brings, by its address. So an install with no trigger that keeps a connection open and no infrastructure up runs nothing between calls and, within Google's and the database's free tiers, costs next to nothing: what it does pay for is storing weft's own images. It is made by the "install on GCP" workflow in the user's own fork of weft. The user owns the GCP project, the billing, the database and the fork; you do the work, through their `gh` and `gcloud`. ## First: the two CLIs, logged in Everything after this step you run yourself, so this step comes first and you do not move on until both work. Check before asking anything: `gh auth status` and `gcloud auth list` (plus `gcloud config get project`). Whatever already works, you skip. - **GitHub.** No account: send them to `https://github.com/signup` and wait. No `gh`: install it for their system (the instructions are at `https://cli.github.com`). Then they run `gh auth login` themselves (it opens a browser; GitHub.com, HTTPS, log in with a web browser), since only they can sign in. Check with `gh auth status`. - **Google Cloud.** No account: send them to `https://console.cloud.google.com`, where they sign in with a Google account, accept the terms and add a billing account (a card is required even for the free tier). No `gcloud`: install it for their system (`https://cloud.google.com/sdk/docs/install`). Then they run `gcloud auth login`. Check with `gcloud auth list`. The only things the user does by hand are what needs them in a browser: signing up, the card, and the two logins. Say each step in plain words and wait for them to say it is done. ## Then you do the install The steps are in the cloud guide (`https://weavemindai.github.io/weft/running/cloud.html`, section "Install weft on GCP"). Run them with the user's CLIs, telling them what you are about to do before anything that creates or costs something: 1. **The project.** A new project made just for weft (weft's own accounts get broad rights inside it, so nothing else should live there): `gcloud projects create <id>`, then link billing (`gcloud billing accounts list`, `gcloud billing projects link <id> --billing-account <account>`). If they already made one for weft, use it. A billing account links only a few projects (five on a new one): if the link fails with `Cloud billing quota exceeded`, offer to reuse a project made for weft before, or to unlink one they no longer use (`gcloud billing projects unlink <id>`), and let them choose. 2. **The region.** If the user is on the free tier and has no reason to be anywhere in particular, the install goes in `us-west1` (zone `us-west1-b`) and its database in Neon's `aws-us-west-2`: both are in Oregon, so every call the install makes to its database stays short. Google's always-free machine and 5 GB of storage exist only in `us-west1`, `us-central1` and `us-east1`, and Neon runs on AWS and Azure, not Google, so that pair is the one place both free tiers sit side by side. If they need another region, take theirs, and the Neon region nearest it. 3. **The fork.** `gh repo fork WeaveMindAI/weft --clone=false`. A new fork has its workflows turned off: turn them on with `gh api -X PUT repos/<fork>/actions/permissions -F enabled=true`. 4. **The `gcloud` block** below, as written, with the three values on its first line filled in (`FORK` is the fork's `owner/repo`, with GitHub's capitals, because Google compares it letter for letter); never invent a flag. Its IAM steps retry on their own while IAM catches up on a brand new project, so a few "waiting for IAM" lines are normal. ```bash PROJECT=my-project REGION=us-west1 FORK=me/weft NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') # On a new project IAM can take a minute to catch up (with the services # just enabled, with the account just made), so each IAM step below is # tried again, quietly, for up to three minutes; a last try shows its # own error. retry() { for _ in $(seq 18); do "$@" >/dev/null 2>&1 && return 0 echo "waiting for IAM to catch up, trying again in 10 seconds" sleep 10 done "$@" } gcloud services enable --project $PROJECT \ iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \ cloudresourcemanager.googleapis.com serviceusage.googleapis.com gcloud storage buckets create gs://$PROJECT-weft-state --project $PROJECT \ --location $REGION --uniform-bucket-level-access gcloud iam service-accounts create weft-installer --project $PROJECT retry gcloud projects add-iam-policy-binding $PROJECT --role roles/owner \ --member serviceAccount:weft-installer@$PROJECT.iam.gserviceaccount.com pool() { gcloud iam workload-identity-pools describe weft-install --project $PROJECT \ --location global >/dev/null 2>&1 \ || gcloud iam workload-identity-pools create weft-install --project $PROJECT \ --location global } provider() { gcloud iam workload-identity-pools providers describe github --project $PROJECT \ --location global --workload-identity-pool weft-install >/dev/null 2>&1 \ || gcloud iam workload-identity-pools providers create-oidc github \ --project $PROJECT --location global --workload-identity-pool weft-install \ --issuer-uri https://token.actions.githubusercontent.com \ --attribute-mapping google.subject=assertion.sub,attribute.repository=assertion.repository \ --attribute-condition "assertion.repository=='$FORK'" } retry pool retry provider retry gcloud iam service-accounts add-iam-policy-binding \ weft-installer@$PROJECT.iam.gserviceaccount.com --project $PROJECT \ --role roles/iam.workloadIdentityUser \ --member principalSet://iam.googleapis.com/projects/$NUMBER/locations/global/workloadIdentityPools/weft-install/attribute.repository/$FORK ``` 5. **The fork's variables**, with `gh variable set <NAME> --repo <fork> --body <value>`: `GCP_PROJECT_ID`, `GCP_REGION`, `GCP_ZONE`, `TF_STATE_BUCKET`, `GCP_WORKLOAD_IDENTITY_PROVIDER` and `GCP_INSTALL_SERVICE_ACCOUNT`. A project's frontend is added later, from the project (read weft-deploying), never here. 6. **The database.** If the user has no Postgres, read weft-database. If they already have a Postgres they want, put its address in the `WEFT_DATABASE_URL` secret with `gh secret set WEFT_DATABASE_URL --repo <fork>` (piping it in, never echoing it into the chat), and when that address goes through a pooler, a direct address of the same database in `WEFT_DATABASE_LISTEN_URL`. 7. **The workflow.** `gh workflow run "install on GCP" --repo <fork>`, then follow it with `gh run watch` in the background. It runs for a while. Its last step prints the install's address (a Cloud Run address, `https://weft-role-dispatcher-<number>.<region>.run.app`, working at once) and the line that logs in with the first operator key, which you read with `gh run view <id> --log` (plain `gh run view` leaves it out). Run that line from the project, after `weft target add` (read weft-deploying): it reads the key from Secret Manager straight into `weft login --key-stdin`, so the key never passes through the chat or a file. Never print the key itself. The install emails when weft's own work is stuck (what `weft status --on prod` lists under `waiting on weft:`). It writes to the fork's repository secret `WEFT_ALERT_EMAIL` if there is one, and otherwise to the first person, by address, holding the Owner role directly on the project. If the user wants the emails somewhere else, set that secret before running the workflow (`gh secret set WEFT_ALERT_EMAIL --repo <fork> --body <address>`). If the workflow stops in "create the cloud" with `no email to send the alerts to was found`, neither exists: set the secret, or, if the user wants no emails, set `alerts_enabled` to `false` in `deploy/terraform/gcp/variables.tf`, then run the workflow again. `alert_after_minutes` in the same file is how many minutes weft keeps retrying before it emails (5 by default). For the rest (what triggers an email, how often, what it costs), go and read the cloud guide's section "What you get". The install needs no domain: its own address is HTTPS and free. A domain costs money, so it is never part of the install; read weft-deploying when the user asks for one. Upgrading weft on the cloud is merging upstream into the fork (`gh repo sync <fork>`) and running the workflow again, then rebuilding the user's CLI from that commit (`./setup.sh --cli`). Each program on it with infrastructure then needs `weft infra upgrade --on prod`, through the deployer: the new weft builds each piece's image anew, and `weft status --on prod` shows `infra: source has changed` until it is done. A program's deploy workflow needs nothing: it asks the install which weft it runs on every run. Sizing (how many triggers one holder takes, a holder's CPU and memory, how many builds run at once, the machine builds compile on) is a default in `deploy/terraform/gcp/variables.tf` in the fork: change it there, commit, and run the workflow again. The build machine defaults to the one Cloud Build's free minutes cover; a bigger one (`build_machine = "E2_HIGHCPU_8"`) compiles a program's image faster and is paid by the minute, once per version of the program, never per worker. Removing weft from GCP is deleting the project made for it, since weft and everything it started live there and nowhere else: `gcloud billing projects unlink <id>`, then `gcloud projects delete <id>` (Google keeps it 30 days, `gcloud projects undelete <id>` brings it back). It deletes everything in it, so say so and wait for the user's yes. The database is theirs, at its own provider, and stays. For deploying projects to the install once the key is in `weft login`, read weft-deploying.