# OpenBKN Deploy
[ไธญๆ](README.zh.md) | English
One-click deployment of **OpenBKN** onto a single-node Kubernetes cluster.
This repository provides scripts to install OpenBKN along with its dependencies including Kubernetes, infrastructure services, and data services.
For portable MariaDB and OpenSearch backups, restore commands, and initial snapshot directory setup, see the [backup and restore guide (ไธญๆ)](BACKUP.zh.md). Entry points: `deploy.sh backup check|run` and `deploy.sh restore check|run`.
**Platforms:** **Linux** is the recommended and fully documented install target (`preflight.sh`, k3s or kubeadm, data services). **macOS** is **optional** for **local development only** (Docker + kind + `dev/mac.sh`); see **[Mac install (dev)](dev/README.md)** ([ไธญๆ](dev/README.zh.md)) โ not a substitute for Linux production installs.
[](../LICENSE.txt)
## Linux: default `k8s` (kubeadm) vs optional k3s
**`KUBE_DISTRO` defaults to `k8s`** (package-manager Kubernetes + **kubeadm** on a single node). **k3s** is an optional lighter single-node stack. If you install k3s with `deploy.sh k3s install`, keep **`preflight.sh`** / **`openbkn`** aligned by passing **`--distro=k3s`** **before** the module (or **`export KUBE_DISTRO=k3s`**) โ otherwise `preflight` may flag a k3s/kubeadm mismatch and installs can disagree on bootstrap.
### kubeadm / `KUBE_DISTRO=k8s` (default)
Single-node kubeadm flow is **`bash ./deploy.sh k8s install`** (`deploy/scripts/services/k8s.sh`). Product modules reuse an existing cluster when `kubectl` already works (`ensure_k8s` skips reinstall), then **`ensure_platform_prerequisites`** installs the bundled **data-services** layer (MariaDB, Redis, Kafka, OpenSearch, โฆ) before Core. **macOS kind** skips host kubeadm but **`openbkn install` still runs `ensure_data_services` first** (see macOS section). Legacy **`kubeadm`** is still accepted as an alias for **`k8s`**.
**`deploy.sh` global flags** (`--distro`, `-y`, `--force-upgrade`, `--config`, โฆ) must appear **before** the module name. Correct: `bash ./deploy.sh --distro=k3s openbkn install`. Wrong: `bash ./deploy.sh openbkn install --distro=k3s` (that `--distro` is not read as a global option). Equivalent without moving flags: `export KUBE_DISTRO=k3s` then `bash ./deploy.sh openbkn install`.
```bash
bash ./deploy.sh k8s install
bash ./deploy.sh openbkn install
```
### k3s (optional โ lightweight single-node)
Uses the upstream k3s installer (Traefik disabled; this stack still installs **ingress-nginx** for a consistent chart/accessAddress setup). Override `K3S_INSTALL_URL`, `INSTALL_K3S_VERSION`, or `INSTALL_K3S_MIRROR` if you need a mirror or air-gapped tuning.
```bash
cd openbkn-deploy
bash ./deploy.sh k3s install
# Align distro with k3s for preflight and platform bootstrap:
bash ./deploy.sh --distro=k3s openbkn install
# or: export KUBE_DISTRO=k3s && bash ./deploy.sh openbkn install
```
Check status: `bash ./deploy.sh k3s status` โ remove: `bash ./deploy.sh k3s uninstall`.
### `accessAddress` vs Kubernetes API (`kubeconfig`)
**`accessAddress` in your install config** is the **HTTP(S) base URL** users hit through Ingress (often a public IP or DNS on **80/443**). It does **not** have to match how **`kubectl` / `helm`** talk to the control plane (**`6443`**).
On the **same Linux host as k3s**, use the file **`/etc/rancher/k3s/k3s.yaml`** for **`kubectl`** / **`helm`** (usually **`server: https://127.0.0.1:6443`** after copying to `~/.kube/config` with correct ownership). **Do not** rewrite the API `server:` to your Elastic IP / public address for on-host use unless you mean to call the API through the public interface with **6443** open and matching **tls-san** โ hairpin/NAT setups often cause **`dial tcp โฆ6443: i/o timeout`**. Remote admins may use a reachable `server:` if the API is exposed and trusted.
### macOS (optional โ local dev with kind)
**Use this only for Mac validation; for real installs use Linux above.** Local Kubernetes via **kind** โ no `preflight.sh` / `k3s` on the Mac host. **`mac.sh` sets `OPENBKN_SKIP_PLATFORM_BOOTSTRAP`** (no host k3s/kubeadm bootstrap). **`openbkn install` now runs `ensure_data_services` first** โ same Helm layer as **`data-services install`** (MariaDB, Redis, Kafka, OpenSearch); **`mac.sh` defaults `AUTO_INSTALL_INGRESS_NGINX=false`** so kindโs existing ingress is not duplicated. Set **`OPENBKN_SKIP_DATA_SERVICES_BUNDLE=true`** to skip bundled data installs (advanced / external infra). **`data-services install`** alone remains useful to pre-stage or refresh the data layer. **Apple Silicon:** kind nodes are **arm64**; use arm64/multi-arch images (see `dev/conf/mac-config.yaml`). **Step order:** [dev/README.md](dev/README.md).
```bash
cd openbkn-deploy # repository root
bash ./dev/mac.sh doctor
# optional: install missing tools via Homebrew โ bash ./dev/mac.sh doctor --fix (or -y doctor --fix to skip confirm)
bash ./dev/mac.sh cluster up
bash ./dev/mac.sh openbkn install # full stack incl. mandatory bkn-safe (auth on); bundled data-services first (same as data-services install)
# optional: bash ./dev/mac.sh data-services install # only if you want the data layer without Core, or to refresh it
# optional: bash ./dev/mac.sh openbkn download
# optional: bash ./dev/mac.sh onboard
# add leading -y for non-interactive (deploy.sh / onboard)
```
Config defaults: `dev/conf/mac-config.yaml`. `isf` is delegated to `deploy.sh` โ see [dev/README.md](dev/README.md).
## ๐ Quick Start
### Host prerequisites
Run install commands as `root` or through `sudo`.
```bash
# 1. Disable firewall
systemctl stop firewalld && systemctl disable firewalld
# 2. Disable swap
swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab
# 3. Set SELinux to permissive if needed
setenforce 0
# 4. Install containerd.io
dnf install containerd.io
```
### Keep firewalld enabled (optional)
The default flow keeps the compatibility mode that disables the host firewall. To retain or re-enable `firewalld`, explicitly enable OpenBKN firewall reconciliation before installation. The script starts and enables `firewalld`, opens only the live ingress HTTP/HTTPS ports, places the live Pod/Service CIDRs and discovered CNI interfaces in the internal `trusted` zone, and leaves Kubernetes API port `6443` closed to the public network by default.
```bash
export OPENBKN_FIREWALL_ENABLED=true
# Set only when remote Kubernetes administration is required; separate CIDRs with commas.
# export K8S_API_ALLOWED_CIDRS="10.10.0.0/16,192.168.1.50/32"
bash ./deploy.sh openbkn install
```
Ingress ports are read from the installed controller's `hostPort` or `nodePort`, so custom `INGRESS_NGINX_HTTP_PORT` and `INGRESS_NGINX_HTTPS_PORT` values are reconciled on every install or upgrade.
The enabled state is persisted at `/etc/openbkn-deploy/firewall-enabled`. Later upgrades continue reconciling OpenBKN firewall rules even when `OPENBKN_FIREWALL_ENABLED` is omitted. An explicit `OPENBKN_FIREWALL_ENABLED=false` skips firewall management for that invocation. Use `OPENBKN_FIREWALL_ENABLED=forget` to permanently stop OpenBKN management without stopping firewalld or deleting existing rules.
### Install OpenBKN
```bash
# 1. Clone the repository
git clone https://github.com/openbkn-ai/deploy.git openbkn-deploy
cd openbkn-deploy
# 2. (Recommended) Pre-install host check / fix
sudo bash ./preflight.sh # check-only (default)
sudo bash ./preflight.sh --fix # check + interactive fixes
sudo bash ./preflight.sh --fix -y # auto-approve every fix
sudo bash ./preflight.sh --list-fixes # preview which fixes would run, no changes
sudo bash ./preflight.sh --help # all flags (--role, --skip, --report, --output=json, โฆ)
# Default checks match k8s/kubeadm; for single-node k3s use: sudo bash ./preflight.sh --distro=k3s
# (same as env KUBE_DISTRO=k3s โ shared with deploy.sh)
# 3. Install OpenBKN (this also installs the install-status dashboard)
# Container-log viewing is enabled by default. The command waits until
# install-status is ready before it succeeds.
sudo bash ./deploy.sh openbkn install
# Default is kubeadm (k8s). For single-node k3s instead (--distro must appear before openbkn):
# sudo bash ./deploy.sh --distro=k3s openbkn install
# or: export KUBE_DISTRO=k3s, then run the installation command above
# The script will interactively prompt for the access address and auto-detect the API server address.
# Or specify addresses explicitly (skips interactive prompts):
# --access_address Address for clients to reach OpenBKN services (can be IP or domain)
# --api_server_address IP bound to a local network interface for K8s API server (must be a real NIC IP)
sudo bash ./deploy.sh openbkn install \
--access_address=<your-ip> \
--api_server_address=<your-ip>
# If you include a port in --access_address, that port is used for the matching
# ingress protocol when the script installs ingress-nginx (for example, https://...:8443
# updates the HTTPS ingress port, while http://...:8080 updates the HTTP port).
# (Optional) Customize ingress ports (default 80/443):
export INGRESS_NGINX_HTTP_PORT=8080
export INGRESS_NGINX_HTTPS_PORT=8443
# Re-run the same install command after changing these values; the script now
# detects port drift and upgrades the existing ingress-nginx release.
# 4. (Recommended) Post-install bootstrap
# Registers an LLM + embedding (skips when already there), patches the BKN ConfigMap
# only when the default actually changes, and on a full (bkn-safe) install creates the business
# user `test`, assigns every role from `openbkn admin role list`, switches `openbkn` to
# that user.
sudo bash ./onboard.sh # interactive
sudo bash ./onboard.sh -y # non-interactive (uses defaults)
sudo bash ./onboard.sh --help # all flags (--config=models.yaml, --enable-bkn-search, โฆ)
```
> After service installation, deploy automatically registers the bundled MinIO bucket `bkn-oss` as the default OSS storage (`minio-default`). This step needs Kubernetes access, not a platform admin login, and preserves an existing enabled default. Initialization failures produce warnings and a retry command without failing the platform installation. Onboard retries the same initialization. Override with `OSS_DEFAULT_BUCKET` / `OSS_DEFAULT_STORAGE_NAME`; set `ONBOARD_SKIP_OSS_STORAGE=true` to skip it in both flows. Kubernetes readiness waits default to 120s each (`OSS_STORAGE_READY_TIMEOUT`).
> **Why `sudo`?** `onboard.sh` reads `$HOME/.openbkn-ai/config.yaml` (written by `sudo deploy.sh` into `/root/.openbkn-ai/`) and writes the `bkn` auth token to `$HOME/.bkn`. Running it without `sudo` falls back to the in-repo template `conf/config.yaml` and may resolve a different access URL. **macOS dev path** (`bash ./dev/mac.sh onboard`) does **not** need `sudo`. The script also prints this hint at startup; silence with `ONBOARD_SUDO_HINT_DISABLED=1`.
> Full preflight / onboard flow and Mermaid diagrams: see [help/en/install.md โ Post-install: `onboard.sh`](../help/en/install.md#post-install-onboardsh).
> **`onboard.sh` runtime messages** are **English**. The seeded admin must change its password on first login; onboard clears that automatically (self-service `/api/safe/v1/auth/change-password`) before the credential login. Chinese + English context: [`dev/README.md`](../dev/README.md#onboard-and-openbkn-full-install) ยท [`dev/README.zh.md`](../dev/README.zh.md); product docs [`help/zh/install.md`](../help/zh/install.md) / [`help/en/install.md`](../help/en/install.md).
### Dev/test: pick chart versions (`--version_file`)
Release installs pin exact chart versions in a committed manifest
(`release-manifests/0.1.5/openbkn.yaml`) โ a lockfile, reproducible.
For **dev/test** you usually want the newest builds, and CI only republishes the
components a branch actually changed. `scripts/gen-dev-manifest.sh` resolves each
chart's version from GHCR and writes a manifest you pass with `--version_file`:
```bash
# latest stable โ every chart = highest clean semver (e.g. 0.1.0)
./scripts/gen-dev-manifest.sh --out=/tmp/m.yaml
# test a branch โ components it rebuilt use the branch build; the rest fall back
# to latest stable, then to the --base branch (default: main)
./scripts/gen-dev-manifest.sh --branch=fix/my-thing --out=/tmp/m.yaml
# install the generated manifest
sudo bash ./deploy.sh --distro=k3s openbkn install --version_file=/tmp/m.yaml
```
An exact chart version in the composition template is preserved and must exist
in GHCR. The `main` placeholder uses the resolution rules below. The sample
service is pinned to `0.1.0-main.20261013062625.shaafa25ac`, which includes dynamic
sample uninstall (#125, backported to 0.2.0 by #130). Its chart pins the service
image by digest. Keep this component version when assembling the 0.2.0 release
lockfile; refreshing the sample catalog does not update the service image.
Per-chart resolution for placeholders (stable-first): `--branch` newest build โ latest stable โ
`--base` newest build โ error. The generated manifest annotates each chart's
source (`branch` / `stable` / `base`). A release branch starts with a full build;
later builds update only the charts that changed. Requires `python3`; see
`./scripts/gen-dev-manifest.sh -h`.
To install the **newest builds for the checked-out branch**, use `--latest` from
`main` or `release/X.Y.Z`. On a release branch it selects each chart's newest
`X.Y.Z-release.<date>.shaโฆ` build, then falls back to stable and main for charts
without a release build. Other branches and detached HEAD are rejected:
```bash
./scripts/gen-dev-manifest.sh --latest --out=/tmp/m.yaml
```
> macOS note: the system `python3` may lack CA certs and silently resolve every
> chart as `NOT FOUND`. Set `SSL_CERT_FILE=/etc/ssl/cert.pem` (or `pip install certifi`).
> `--latest` reads the branch of this repository's checkout. Run it from a
> `main` or `release/X.Y.Z` checkout.
### Install behind a restricted network (CN / no docker.io / slow GHCR)
On clusters that can't reach `docker.io` or pull GHCR image blobs (read timeouts),
`openbkn install` accepts three flags so the **script** handles it โ no manual
`crictl pull`/retag:
- **`--registry=<swr / ghcr / host/ns>`** โ image registry for **BKN images** and the bundled **data-service / ingress** images (sugar for `--set image.registry`). `swr` โ `swr.cn-east-3.myhuaweicloud.com/openbkn-ai`, `ghcr` โ `ghcr.io/openbkn-ai`. Precedence: explicit `--set image.registry=โฆ` > `--registry` > an `image.registry` already in your `--config` YAML (respected, e.g. `dev/conf/mac-config.yaml`) > default `swr` (when config file does not set `image.registry`). SWR mirrors the same `โฆ-main.<date>.shaโฆ` build tags as GHCR.
- **`--dockerhub-mirror=<auto / host / off>`** โ containerd `docker.io` mirror for **third-party images** (otel/hydra/postgres/minio). Writes `/etc/containerd/certs.d/docker.io/hosts.toml` (needs root + a containerd `config_path` certs.d; else it warns and skips, never fails). **Defaults to `auto`** โ probes a candidate list and picks the first mirror that serves this stack's docker.io images over the mirror (`?ns=docker.io`) protocol (sentinel `oryd/hydra`; `docker.m.daocloud.io` 403s namespaced repos there, so a fixed default isn't safe). Pass a host to pin one (e.g. `docker.1panel.live`); `off` disables. Candidate list overridable via `OPENBKN_DOCKERHUB_MIRROR_CANDIDATES`.
- **`--latest`** โ when no `--version_file` is given, installs the newest builds for this checkout's `main` or `release/X.Y.Z` branch (requires `git`).
```bash
# Newest builds + BKN images from SWR + docker.io third-party via the default mirror:
sudo bash ./deploy.sh openbkn install --latest --registry=swr
# Or with a pre-generated manifest (e.g. built on a dev box, no git on the target):
sudo bash ./deploy.sh openbkn install --version_file=/tmp/m.yaml --registry=swr
```
> Moving BKN images into an air-gapped cluster by hand: versions published from
> 0.1.5 on also carry single-architecture tags, `<image>:<version>-amd64` and
> `<image>:<version>-arm64` (earlier versions have only the multi-arch
> `<image>:<version>`). Pull the one matching the target node on any machine โ
> no `--platform`, no skopeo โ retag it to the plain version, and import it
> into the node's containerd (`k3s ctr` on k3s):
>
> ```bash
> docker pull ghcr.io/openbkn-ai/bkn-backend:0.1.5-arm64
> docker tag ghcr.io/openbkn-ai/bkn-backend:0.1.5-arm64 ghcr.io/openbkn-ai/bkn-backend:0.1.5
> docker save ghcr.io/openbkn-ai/bkn-backend:0.1.5 -o bkn-backend.tar
> # on the target node:
> ctr -n k8s.io images import bkn-backend.tar
> ```
> The committed migrations fix any DB-schema drift (e.g. `vega-backend` 0.9.x), but
> only run when the **data-migrator pre-install job** runs โ i.e. via `openbkn install`,
> not a bare `kubectl set image`.
### Resource Configuration
You can set Kubernetes resource requests and limits for all Core services via environment variables:
```bash
# Set CPU and memory requests
OPENBKN_CORE_REQ_CPU=200m OPENBKN_CORE_REQ_MEM=512Mi \
sudo bash ./deploy.sh openbkn install
# Set full resource limits
OPENBKN_CORE_REQ_CPU=200m OPENBKN_CORE_REQ_MEM=512Mi \
OPENBKN_CORE_LIM_CPU=2 OPENBKN_CORE_LIM_MEM=2Gi \
sudo bash ./deploy.sh openbkn install
```
| Environment Variable | Description | Example Values |
| --- | --- | --- |
| `OPENBKN_CORE_REQ_CPU` | CPU request value | `200m`, `1` |
| `OPENBKN_CORE_REQ_MEM` | Memory request value | `512Mi`, `1Gi` |
| `OPENBKN_CORE_LIM_CPU` | CPU limit value | `2`, `4` |
| `OPENBKN_CORE_LIM_MEM` | Memory limit value | `2Gi`, `4Gi` |
> **Notes**:
> - These variables default to empty; chart defaults are used when unset
> - In k3s mode, lightweight resource defaults are applied (CPU request 100m, memory request 128Mi); override with the variables above
> - Settings apply uniformly to all bkn-core releases
## ๐ Prerequisites
### System requirements
| Item | Minimum configuration (testing and learning) | Recommended configuration (minimum for production) |
| --- | --- | --- |
| Operating system | CentOS 8+, OpenEuler 23+, or compatible Linux | CentOS 8+ |
| CPU | 4 cores | 16 cores or more |
| Memory | 8 GB | 32 GB or more |
| Disk | 50 GB | 500 GB or more |
| Privileges | root or a user with sudo access | root or a user with sudo access |
> **Configuration and validation scope**: The minimum configuration is for testing and learning and still requires full deployment and runtime validation; it does not mean the default installation has been verified on that hardware. For constrained hosts, explicitly use `sudo bash ./deploy.sh --distro=k3s openbkn install`; the default kubeadm path retains each chart's resource settings. These requirements do not automatically tune service resources or guarantee capacity under production workloads.
>
> **Preflight measurements**: GB uses decimal units (1 GB = 1,000,000,000 bytes); CPU checks use the logical core count returned by `nproc`. Below minimum is `FAIL`, minimum met but below recommended is `WARN`, and recommended met is `OK`. Disk checks measure **available space** on the filesystems containing `/`, `/var`, and the container image directory, rather than nominal disk capacity; repeated checks of the same filesystem are not additive. Images and data share the disk budget, with no separate 100 GiB image-directory requirement. The 50 GB minimum leaves limited headroom; allow space for the OS, images, data growth, and upgrades.
### Network requirements
The deployment scripts need access to these domains:
| Domain | Purpose |
| --- | --- |
| `mirrors.aliyun.com` | RPM package mirrors |
| `mirrors.tuna.tsinghua.edu.cn` | `containerd.io` RPM mirror |
| `registry.aliyuncs.com` | Kubernetes component images |
| `swr.cn-east-3.myhuaweicloud.com` | OpenBKN application image registry |
| `repo.huaweicloud.com` | Helm binary download |
| `openbkn-ai.github.io` | OPenbkn Helm chart repository |
| `rancher-mirror.rancher.cn` | k3s install script / binary (k3s quickstart path; override with `K3S_INSTALL_URL`) |
## ๐ฆ Deployment Model
`openbkn` is the OpenBKN product-level entrypoint in this repository. The install flow is:
1. Install or repair single-node Kubernetes, local-path storage, and ingress-nginx.
2. Install or repair data services: MariaDB, Redis, Kafka, and OpenSearch.
3. Deploy the OpenBKN application charts.
The Core application layer includes charts for data services management, application deployment, and task orchestration.
## ๐ง Usage
### Recommended commands
```bash
# Install OpenBKN
./deploy.sh openbkn install
# Show Core status
./deploy.sh openbkn status
# Uninstall Core
./deploy.sh openbkn uninstall
# Cluster and Pod status
kubectl get nodes
kubectl get pods -A
```
### Install status & health
`openbkn status` prints a live, detailed table for operators on the server โ for
every release in the manifest it shows the **expected vs deployed** chart version,
app version, helm revision/status, workload ready count (`DRIFT`/`MISSING`
flagged), the bundled dependency services, and a per-service **application health**
section (probed via the apiserver service proxy: `/health/ready` โ `/api/v1/health`
โ `/healthz` โ `/health`, classified `up` / `degraded` / `no-endpoint`).
```bash
# Detailed live status table (versions, ready, drift, service health)
./deploy.sh openbkn status
# (Re)publish the non-sensitive JSON snapshot + the /install-status endpoint
# without reinstalling. Runs automatically at the end of `openbkn install`.
./deploy.sh openbkn publish-status
```
For an offline environment, sync the latest images (including
`openbkn-ai/library/nginx:1.27-alpine`) before republishing the endpoint, and
pass the offline flag to `publish-status`:
```bash
./scripts/sync-k8s-images.sh <offline-registry>
./deploy.sh --offline=<offline-registry> openbkn publish-status
```
A dashboard is served through the administrator-only ingress/port (a tiny nginx
serving ConfigMaps โ see `conf/install-status/`). The administrator entry point
redirects with a Studio access token in the URL fragment; there is no manual
token-entry form:
- `GET /install-status` โ an HTML page rendering releases, per-service health, and
dependency topology (auto-refreshes; static, no build step / CDN).
- `GET /install-status.json` โ the raw JSON snapshot the page consumes (admin token required).
```bash
# Administrator redirect target: https://<access-address>/install-status#token=<Studio-access-token>
# English UI: https://<access-address>/install-status?lang=en
# Chinese UI: https://<access-address>/install-status?lang=zh (default)
```
The token is read from the URL fragment and removed from the address bar
immediately. A tab-scoped `sessionStorage` copy permits refresh for up to 15
minutes while the same-origin Studio access-token cookie still matches.
Studio logout, cookie rotation, expiry, or a 401/403 response clears the copy.
This dashboard requires Studio and `/install-status` on the same origin. Reopen Service
operations from the Studio administrator menu when authorization is required.
Every status and log request still uses the bkn-safe administrator check.
The existing `/install-status` page now shows whitelisted container waiting and
termination reasons, exit codes, rollout counts for Deployments, StatefulSets,
DaemonSets and Jobs, safe Warning Event summaries, and probe-method summaries.
Each Pod also has an expandable sanitized Describe summary covering lifecycle,
Pod Conditions, init containers, container state, and related Warning Events;
init-container logs are available from that summary. It excludes nodes/IPs,
annotations, environment, mounts, and raw Event messages. Version, dependency,
and HTTP-health data comes from the latest publication; Pod, workload, and
Event data refreshes every 30 seconds.
It displays the last successful collection time for Pod, workload and Event
snapshots, flags data older than 90 seconds, and distinguishes chart versions
from the live image-tag source. The redirect token is read from the URL fragment
(which is never sent to nginx), immediately removed from the address bar, and
kept only in page memory. All snapshot, Pod, workload, Event and log endpoints
independently validate it through bkn-safe's Studio `is_admin` rule. Logs are
never fetched automatically or included in public JSON. Clicking a log button
opens a modal dialog, where operators can switch between recent and previous
logs or search loaded lines; closing it clears the displayed log content. The
reader returns at most 200 lines / 64 KiB; page links request 100 lines. Access
audit entries (Studio account, target, line bound and result) are written to the
reader container log without log content or credentials; inspect them with
`kubectl -n <namespace> logs deployment/install-status -c log-reader`.
Log access is enabled by default and requires the same administrator Token.
Application logs can contain secrets that the status service cannot reliably
redact; disable it and republish where this exposure is unacceptable:
```bash
INSTALL_STATUS_LOGS_ENABLED=false ./deploy.sh openbkn publish-status
```
Only the existing `install-status` Deployment and Service are used; there is no
second workload or public operations API. A Python reader container is added to
the same Pod, using `swr.cn-east-3.myhuaweicloud.com/openbkn-ai/library/python:3.12-alpine`
online. Sync this
image for offline deployment or override it with `INSTALL_STATUS_PYTHON_IMAGE`.
Because this is a single-Pod design, an image pull failure can block the whole
Deployment rollout. This change also grants namespace-scoped `pods/log get` to
the existing ServiceAccount, and read access to Events, Deployments,
StatefulSets, DaemonSets and Jobs in the namespace.
The authenticated status view carries product/release versions, ready counts, dependency-service connection
topology, and classified per-service health โ and **deliberately no credentials**:
the collector ([scripts/lib/install_status.py](scripts/lib/install_status.py))
whitelists fields (host/port/type only; passwords, users, keys, tokens are
dropped) and never exposes raw health-endpoint bodies (those can carry internal
versions/topology). The ConfigMap is refreshed per install; nginx reads the
mounted file per request, so no pod restart is needed.
### Studio one-click samples
The standard product composition includes `bkn-sample-studio`. The installer supplies the existing internal ingress-nginx HTTP gateway through the Chart `baseURL` value, independently of the public HTTPS access address; explicit operator `--set` overrides remain supported. Incorrect stored sample gateway values are reconciled even when the Chart version is unchanged. Published sample Charts pin the service/executor image digest. The installer does not infer API compatibility from Chart versions or component names.
With the sample service's standard-API runtime defaults, leave `platform.version` and `platform.capabilities` empty: the service reads backend version through `GET /api/bkn-backend/v1/health` and checks required API adapter names against its implemented installation profile. Target API availability, authorization and licensing are enforced by the actual installation calls, with failures recorded in the corresponding stages. `GET /api/safe/v1/capabilities` describes licensed capabilities and is not the sample API inventory. Older sample service builds still require explicit compatibility values; use a published build containing the runtime-defaults change. See the [sample service deployment guide](https://github.com/openbkn-ai/bkn-samples/blob/main/docs/sample-service-deployment.md). Full fresh-product installation acceptance remains separate from configuration/preflight checks.
After signing in, open Studio Home โ One-click experience. Click **Refresh samples** to scan official releases, inspect a version's release notes and install it. Without an online catalog, administrators can **Import offline package**; its fixed data image must be available to the cluster. The internal `openbkn-samples` Kubernetes namespace is managed by the Chart. The current scope is one OpenBKN instance per environment; in-place sample upgrades and uninstall are not supported. See the [sample development guide](https://github.com/openbkn-ai/bkn-samples/blob/main/docs/SAMPLE_DEVELOPMENT_GUIDE.md).
## ๐ Project Structure
```text
deploy/
โโโ deploy.sh # Main entry script
โโโ conf/ # Bundled config and static manifests
โ โโโ install-status/ # /install-status endpoint manifests (nginx + ingress)
โโโ release-manifests/ # Versioned release bill of materials
โโโ scripts/
โ โโโ lib/ # Common helper functions (install_status.py: status collector)
โ โโโ services/ # Product and dependency install scripts (status.sh: install status)
โ โโโ sql/ # Versioned SQL initialization scripts
โโโ .tmp/charts/ # Local chart cache generated by download
```
## ๐๏ธ Uninstall
`bash deploy.sh openbkn uninstall` removes only the Core application layer.
```bash
# Remove the Core application layer
./deploy.sh openbkn uninstall
```
`bash deploy.sh k8s reset` resets the Kubernetes cluster, including data services and core.
```bash
# Reset Kubernetes cluster
./deploy.sh k8s reset
```
## ๐ Troubleshooting
### CoreDNS is not ready
```bash
# Check whether firewall is disabled
systemctl status firewalld
# Restart CoreDNS
kubectl -n kube-system delete pod -l k8s-app=kube-dns
```
### Pods fail to pull images
```bash
# Check network connectivity
curl -I https://swr.cn-east-3.myhuaweicloud.com
# Check containerd config
cat /etc/containerd/config.toml
```
### Kubernetes apt / yum source missing or 404
`preflight.sh --check-only` reports one of these in **strict mode** (default):
```text
[FAIL] Deprecated Kubernetes apt source detected (packages.cloud.google.com) ...
[FAIL] apt has no install candidate for kubeadm โ Kubernetes apt source missing or unreachable.
[FAIL] dnf/yum has no install candidate for kubeadm โ Kubernetes yum repo missing or unreachable.
```
**Recommended fix (one command):**
```bash
sudo bash deploy/preflight.sh --fix --fix-allow=k8s-pkgs-repo
# or, to also pre-stage containerd / helm / Node, etc.:
sudo bash deploy/preflight.sh --fix -y
```
`preflight --fix โ k8s-pkgs-repo` covers **both** scenarios (legacy alias `k8s-apt-source` still works in `--fix-allow`):
- Legacy `packages.cloud.google.com` source present โ migrate it to `pkgs.k8s.io`.
- No K8s source configured at all โ write `/etc/apt/sources.list.d/kubernetes.list` (or `/etc/yum.repos.d/kubernetes.repo`) pointing to `pkgs.k8s.io/core:/stable:/<vX.Y>/deb|rpm/`.
Pin a specific minor with `PREFLIGHT_K8S_APT_MINOR=v1.28` (default: detected from installed `kubeadm`, falls back to `v1.28`).
**Manual fallback (Ubuntu/Debian):**
```bash
sudo apt-mark unhold kubeadm kubelet kubectl || true
sudo apt remove -y kubeadm kubelet kubectl
sudo rm -f /etc/apt/sources.list.d/kubernetes.list
sudo rm -f /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key \
| sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' \
| sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
```
**Manual fallback (RHEL/CentOS/openEuler):**
```bash
sudo tee /etc/yum.repos.d/kubernetes.repo > /dev/null <<'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
sudo dnf install -y --disableexcludes=kubernetes kubeadm kubelet kubectl # or: yum
```
### `containerd` cannot be installed (no `containerd.io` candidate)
On stock Ubuntu (no Docker CE repo) `preflight.sh` reports:
```text
[FAIL] apt has no install candidate for containerd.io OR containerd.
[FAIL] containerd not found ...
```
`preflight --fix โ containerd-install` now tries `containerd.io` first (Docker CE repo) and falls back to the distro `containerd` package automatically:
```bash
sudo bash deploy/preflight.sh --fix --fix-allow=containerd-install
```
If both fail, the apt/yum source itself is broken โ fix `apt-get update` / `dnf repolist` first.
### Strict mode and `--lenient`
`preflight.sh` defaults to **strict mode** (`PREFLIGHT_STRICT=true`). Items that block install AND are auto-fixable by `--fix` are reported as `[FAIL]` (so `--check-only` exits `1`), not `[WARN]`:
- `swap`, `net.ipv4.ip_forward`, `br_netfilter` / `overlay` modules, `bridge-nf-call-*`
- `vm.max_map_count`, `fs.inotify.*`, `ulimit -n soft`
- `containerd` not found / socket missing, `kubectl`, `helm`, `overlay` fs
- broken `apt-get update`, `dnf/yum repolist` failures, missing kubeadm/containerd install candidate
To downgrade these back to `[WARN]` (e.g. on a low-spec lab box where you accept the risk):
```bash
sudo bash deploy/preflight.sh --check-only --lenient
# equivalent to: PREFLIGHT_STRICT=false PREFLIGHT_STRICT_SOURCES=false sudo bash deploy/preflight.sh
```
### View component logs
```bash
kubectl logs -n <namespace> <pod-name>
```
### OAuth login redirect_uri mismatch
Login fails with `invalid_request ... 'redirect_uri' ... does not match ... pre-registered redirect urls`:
the callback the browser sent is not in the login client's (`openbkn-studio`, etc.)
registered list. These clients live in Hydra and are registered by the bkn-safe
chart's client-seed Job from `accessAddress` and `clientSeed.*`. The callback is the
browser's address + `/studio/callback` (e.g. `https://<access-address>/studio/callback`).
```bash
# 1. Standard dev port (least effort): run the local studio dev server on
# localhost:8000; http://localhost:8000/studio/callback is registered by default.
# 2. Permanent / production address -> add to chart values, then reinstall (the
# seed Job re-registers it and it survives upgrades):
# add an entry to clientSeed.extraWebRedirectUris in
# bkn-safe/charts/bkn-safe/values.yaml. The server-side studio's gateway callback
# is derived from accessAddress automatically โ no need to list it.
# 3. Temporary / dev address -> no reinstall, add via the bkn-safe admin API
# (super-admin required; wiped on the next upgrade):
export BKN_HOST=https://<access-address>
openbkn auth login "$BKN_HOST" # must be a super-admin session
bash deploy/scripts/bkn-redirect.sh add http://localhost:5173/studio/callback
bash deploy/scripts/bkn-redirect.sh list
bash deploy/scripts/bkn-redirect.sh del http://localhost:5173/studio/callback
```
See [bkn-safe/docs/oauth-redirect-uris.md](../bkn-safe/docs/oauth-redirect-uris.md) for detail.
## ๐ License
[Apache License 2.0](../LICENSE)
## Internal Trace Admission protocol upgrade
This install profile targets the protocol in Foundry PR #2047. Policy, configuration, heartbeat and ACK use `agent-observability-internal:8081` without Admission OAuth credentials or Ed25519 keys. Upgrade the server, publishers, Collector and Charts as a compatible set; there is no legacy client compatibility layer.
The complete install enables five Kafka Evidence publishers. Kafka SASL still requires credentials: the installer copies the first client password from `resource/kafka-sasl` into `bkn-trace-evidence-kafka` in the application namespace, using `KAFKA_CLIENT_USER` as the username. For external Kafka, provision the target Secret with `username` and `password`; `OPENBKN_TRACE_KAFKA_SECRET` overrides its name. Artifact credentials remain separate.
The install also enables agent-observability's Evidence Kafka consumer, using the same broker and credential Secret. A separate Admin Job (96 MiB Java heap, 256 MiB container limit) creates `openbkn.evidence.v1` when absent with `message.timestamp.type=LogAppendTime`, then verifies the effective setting before deploying Core. It never runs Admin tools inside the broker. Existing topics are not altered: a `CreateTime` topic stops installation with an actionable error, because changing the setting does not rewrite old messages. Owners must review the migration and then verify newly produced MCP Evidence. The Job is cleaned up on success or failure.
For external Kafka, the client needs permission to create/describe this topic, or the operator must provision it first. `OPENBKN_TRACE_KAFKA_ADMIN_IMAGE` selects a Kafka CLI image; the default uses the bundled Kafka image. SASL defaults to PLAIN; use `OPENBKN_TRACE_KAFKA_SASL_MECHANISM` for SCRAM-SHA-256 or SCRAM-SHA-512. Topic creation defaults to one partition and replication factor one, matching the bundled broker; override `OPENBKN_TRACE_TOPIC_PARTITIONS` and `OPENBKN_TRACE_TOPIC_REPLICATION_FACTOR` for a larger cluster. An explicit `--set kafkaConsumers.evidence.enabled=false` skips the topic check. The indexed broker, Secret name and SASL `--set` overrides are also honored by the check.
Before upgrading, review `networkPolicy.allowedClients` for all six callers and the actual namespace/Pod labels. The resource-limit cleanup path exports old effective Helm values, so an old allowlist can survive even without `--reuse-values`. Explicitly update it. Same-version installs reconcile missing or outdated publisher/control values.
The default kubeadm installer uses Flannel, whose data plane does not enforce NetworkPolicy by itself. Configure a policy enforcement component and verify it in the target environment. See https://github.com/flannel-io/flannel/blob/master/Documentation/netpol.md . Verify enforcement on k3s and other distributions as well.
Acceptance requires all six callers to read policy/configuration and send heartbeat/ACK, an unlisted Pod to fail access to port 8081, public internal routes to return 404, and an enable/disable operation to converge with Evidence delivery. Test fresh installation and existing-version upgrades before closing Foundry #2043/#2046. Roll back the server, clients, Collector and Charts together, restoring the OAuth/signing configuration required by the old version.
Cluster-free regressions:
```bash
bash scripts/services/openbkn_trace_control_test.sh
bash scripts/services/openbkn_trace_profile_test.sh
bash scripts/services/openbkn_trace_prerequisites_test.sh
bash scripts/services/openbkn_trace_topic_test.sh
python3 scripts/lib/trace_kafka_topic_job_test.py
```
Set `FOUNDRY_ROOT` to a compatible checkout for additional index and data-migrator contract checks. A standalone checkout explicitly reports those cross-repository checks as skipped. These checks do not replace Chart rendering or cluster acceptance.
## System Audit installation (Foundry #2061 / #2094)
The complete installer also enables Safe Audit/access publishing with the configured MQ broker and existing Kafka credential Secret. `auditPublisher.environment` uses current config/CLI, then the saved Safe environment, then the saved AO environment (default for a new deployment: `development`); Safe uses the flat `credentialsSecretName`, `usernameSecretKey` and `passwordSecretKey` fields. Missing/disabled publisher configuration is repaired even when the Safe Chart version is unchanged; a matching configuration skips the rollout. Verify new user-management and login/logout events through Kafka, Ledger and the log API/Studio; Pod Ready or historical records alone do not establish coverage.
This ports the installer wiring from Foundry PR #2097, not application code or Charts. The standalone Chart default `kafkaConsumers.audit.enabled=false` stays unchanged. The complete installer explicitly enables the Audit consumer and Core publisher in final **agent-observability** values and uses Core MariaDB with migration enabled. The default independent consumer group is `bkn-trace-audit-ledger-v1`; existing group and Secret references are preserved.
No extra `--config` file is required. Existing Trace prerequisites prepare `bkn-trace-core-mariadb` and reuse `bkn-trace-evidence-kafka` (`OPENBKN_TRACE_KAFKA_SECRET` overrides the Kafka Secret name). Brokers and SASL come from the existing MQ configuration. External Kafka still requires an existing credentials Secret. No Kafka users, passwords, ACLs, or broker authentication settings are changed.
Publisher environment defaults to `development`; a saved valid value or an explicit override is reused. No environment argument is required for the default installation on `main`:
```bash
./deploy.sh openbkn install --registry=swr --latest \
--set resources.requests.cpu=0m --set resources.requests.memory=0Mi
```
The consumer and publisher may reuse the existing Kafka credentials Secret; there is no production-only requirement for a separate approved write Secret. Override `auditPublisher.environment` or `auditPublisher.existingSecret.name` only when needed. No account is provisioned automatically. Both components support `existingSecret.usernameKey` / `passwordKey` (defaults: `username` / `password`); credentials use Secret references, not Helm plaintext. The existing publisher requires PLAIN; unsupported mechanisms fail before application upgrades.
After existing Trace prerequisites, before the application release loop, client-only `helm template` validates the staged pinned AO Chart, final merged values, and required Secret keys. A bounded separate Kafka Admin Job checks `openbkn.audit.v1`. Only an absent topic is created with `message.timestamp.type=LogAppendTime`. Existing `CreateTime` stops installation and requires a separately reviewed migration: changing configuration cannot rewrite historical records. Existing topics, records, offsets, authentication and Evidence topic behavior are not modified. The Job is deleted on success/failure.
Same-version Audit repair preserves AO's Chart-owned user settings including image, resources, Evidence, OpenSearch and EE. Unintended non-Audit changes are rejected; current explicit inputs may change them. AO skips only when final configuration and original reconciliation checks match. Pending releases are not automatically uninstalled; concurrent revision changes stop the upgrade. Other release order remains unchanged. Failures return nonzero and stop the application release loop; earlier prerequisites are not automatically rolled back.
Real-cluster acceptance: both Audit flags true, valid environment/Secret references, unchanged Kafka authentication and other service settings, effective `LogAppendTime`, healthy consumer/publisher, and a real audited business operation visible in System Audit without `audit_consumer_not_configured`. An empty HTTP 200 alone is insufficient. Cover fresh install, same-version repair, repeat install and failures. Rollback the prior AO Helm revision only; do not delete database tables, Kafka messages or offsets.
Cluster-free regressions (Helm 3.18.6 and Python 3):
```bash
PYTHONPATH=scripts/lib python3 -m unittest test_audit_consumer test_audit_helm_render audit_kafka_topic_job_test
bash scripts/services/openbkn_audit_consumer_test.sh
```
The small Helm fixture and mocked cluster tests do not replace published product Chart rendering or real-cluster acceptance.
## Trace/Evidence admission budget
Set `observability.admissionBudget` in the persistent `--config` YAML used for installation and upgrades. Config generation preserves the existing `observability` block. New configs supply `collectorMetricsEndpoint: http://otelcol-contrib:8888/metrics` for the bundled Collector; change it when using a different deployment topology. No Admission credentials or permissions are added by this configuration.
Fill all four deployment/SLO thresholds (`opensearchCapacityThreshold`, `opensearchHeapThreshold`, `collectorQueueThreshold`, `storagePoolThreshold`) with finite numbers in `(0, 1]`. They deliberately have no default. Leaving them empty keeps budget-dependent configuration reads and enable requests unavailable, with explicit diagnostic fields; it does not disable existing capture or stop unrelated APIs. The metrics endpoint must provide both `otelcol_exporter_queue_size` and a positive `otelcol_exporter_queue_capacity`.
Keep these values in the config used on every upgrade. A separate one-time Helm override or `kubectl set env` does not establish that persistent configuration source. Add the block explicitly to existing configurations before requesting Trace/Evidence enable. If applying config changes without changing Chart versions, use `--force-upgrade` so the installer does not skip the release.
### Sandbox REST gateway
Sandbox OSDK uses an explicit `controlPlane.env.BKN_BASE_URL` before any installer
discovery. Existing explicit Helm values are preserved; current values files and
`--set` arguments override them. Without an origin, the installer discovers a
single ingress-nginx controller Service and uses its HTTP Service port, not its
NodePort. Its selector/target port configure a separate executor gateway egress
rule. No REST address is derived from the MCP URL and no executor needs cluster
discovery permissions. Missing or ambiguous controllers stop before rollout.
For a custom origin, configure `networkPolicy.gateway` or `additionalEgress` as
needed; discovery is not required when the explicit origin is valid.
This requires the matching Foundry sandbox Chart with `controlPlane.gateway`
support. Existing pooled executors need a controlled refresh with workspace
preservation after configuration rollout. The installer does not restart them
merely by changing a ConfigMap.
Regression checks against the matching Foundry checkout:
```bash
BKN_FOUNDRY_SOURCE=/path/to/bkn-foundry bash scripts/services/openbkn_sandbox_gateway_test.sh
python3 -m unittest discover -s scripts/lib -p test_sandbox_gateway.py
```