kibana-dev-env

name

kibana-dev-env

description

Kibana local development environment manager — tmux sessions, git worktrees, ES + Kibana lifecycle, remote ES credentials, scoped checks, data ingestion, and Synthetics failure scenario injection. Use this skill whenever the developer mentions: starting Kibana, switching branches, managing worktrees, running ES locally or remotely, creating SLOs, setting up synthetics private locations, ingesting test data, breaking/fixing Synthetics state, triggering failure scenarios, running scoped lint/typecheck/jest, checking session health, or anything related to their local Kibana dev workflow. Also trigger when the developer asks about ports, tmux sessions, kibana.dev.yml, remote ES configuration, oblt-cli credentials, or bootstrapping their environment.

Kibana Dev Environment

This skill manages a tmux-based Kibana development environment that supports multiple concurrent branches via git worktrees, each with isolated ES + Kibana instances on separate ports. It handles the full lifecycle: session creation, branch switching, server restart, credential renewal, health checks, data ingestion, and scoped code quality checks.

Installation

The scripts/ folder contains all the scripts and config templates needed to run this environment. All scripts must stay together in the same directory — dev-start.sh locates templates and helpers relative to its own location.

# From the scripts/ folder, make the scripts executable
chmod +x scripts/*.sh

# Create your config (edit paths and ports to match your machine)
cp scripts/kibana-dev.conf.example ~/.kibana-dev.conf

# Run the setup wizard
scripts/dev-start.sh setup

When running commands on behalf of the developer, always use the absolute path to scripts/dev-start.sh based on where this skill is installed. Do not ask the developer to set up aliases or symlinks — just use the full path directly.

Important: always use dev-start.sh

Never suggest raw tmux commands (tmux attach, tmux send-keys, etc.) to the developer. Always use the dev-start.sh CLI — it wraps all tmux operations and handles port assignments, config generation, and process lifecycle automatically. Similarly, kbn-start.sh handles yarn start with the correct port arguments — never suggest running yarn start directly without port flags.

kbn-start.sh also handles Node version switching (nvm use) automatically before starting ES or Kibana. Never tell the developer to run nvm use manually — the scripts take care of it.

Important: never hardcode paths

Always refer to paths using their config variable names (KIBANA_MAIN_DIR, WORKTREE_BASE, etc.) from ~/.kibana-dev.conf. Never hardcode paths like ~/Documents/Development/kibana in responses — the developer's paths may differ.

Important: auto-detect, don't ask

When the developer asks to create resources (SLOs, private locations, etc.) or interact with a running Kibana, read config/kibana.dev.yml and ~/.kibana-dev.conf automatically to get ports, credentials, and ES host. Don't ask the developer to look these up — the config files have everything.

Important: always use ports from config, not defaults

This setup runs multiple Kibana instances on different ports (feat=5601, main=5602, hotfix=5603+). When making Kibana API calls — whether directly or through other skills like observability-manage-slos or kibana-api — always read the port and credentials from config/kibana.dev.yml in the current working directory. Never fall back to hardcoded defaults like localhost:5601 or elastic:changeme.

The kibana-api utility in the Kibana repo only tries localhost:5601 — this is wrong for main and hotfix sessions. This skill's config detection takes precedence: read config/kibana.dev.yml for the correct port, then use it for all API calls.

Architecture

dev-start.sh (CLI router)
  ├── kbn-start.sh    (bootstrap ES + Kibana in tmux panes)
  ├── run-checks.sh   (scoped lint / typecheck / jest)
  └── run-data.sh     (SLO + synthetics data ingestion + failure scenarios + full reset)

All scripts share config from ~/.kibana-dev.conf (paths, ports, cluster name) and per-worktree config/kibana.dev.yml (ES connection, Kibana settings).

Session structure

Each tmux session has these windows:

Window	Purpose
servers	Pane 0: ES (or kbn-start.sh), Pane 1: Kibana (yarn start with port flags)
cursor	IDE MCP config, KIBANA_URL set
scripts	Pre-populated data ingestion commands
git	Git operations, branch management
editor	File editing
checks	Lint / typecheck / jest (full sessions)
ftr	Functional test runner (full sessions)

Lightweight sessions (created with new) have: servers, cursor, scripts, git, editor.

Port assignments

Ports are configurable in ~/.kibana-dev.conf. The defaults are:

• kibana-feat: Kibana 5601, ES 9200 (the "primary" ports — feat is the active dev session)
• kibana-main: Kibana 5602, ES 9201 (secondary, for main branch)
• Hotfix sessions: auto-assigned starting from 5603/9202

Always read ~/.kibana-dev.conf or config/kibana.dev.yml for the actual ports rather than assuming defaults — the developer may have customized them.

Commands

Run all commands as ~/dev-start.sh <command>:

Sessions

Command	What it does
(no args)	Start/attach all sessions + auto-recover orphaned worktrees
switch <branch> [--remote]	Destructive — kills kibana-feat, removes its worktree, creates a new one
new <branch> [--full] [--remote]	Non-destructive — creates an additional session alongside existing ones
attach <branch>	Attach to an existing session
kill <branch>	Kill session + remove worktree
kill-all	Kill ALL kibana-* sessions
recover	Recreate sessions for orphaned worktrees (auto-start ES + Kibana)
prune	Remove orphaned worktrees (no active tmux session)

Choosing new vs switch

Default to new unless the developer explicitly asks to replace the current feat session. switch kills the running kibana-feat session and removes its worktree — this is destructive and the developer may have unsaved work, running servers, or open terminals in that session.

Developer says	Use	Why
"start working on a new branch"	new	Preserves current sessions
"create a session for branch X"	new	Additive, no disruption
"I want to work on feature/X"	new	Ambiguous → default to non-destructive
"switch feat to branch X"	switch	Explicitly asked to replace feat
"replace my current feature branch"	switch	Explicitly asked to replace feat

When in doubt, ask: "Do you want a new session alongside kibana-feat, or replace it?"

Operations

Command	What it does
restart <main|feat|branch>	Restart ES + Kibana (handles graceful shutdown, auto-rebuilds session if panes are missing)
renew [--cluster-name <n>] [--session <t>] [--save]	Refresh remote ES credentials; --session limits to one session (main|feat|branch)
sync [target] [--remote|--local]	Regenerate kibana.dev.yml from template (target: main|feat|branch|all)
clean [main|feat|name|all]	List or delete ES data folders

Info

Command	What it does
list	Sessions, worktrees & port assignments
status	Health check — ping ES + Kibana
setup	Interactive config wizard

Navigating sessions

To switch between sessions and windows, the developer uses tmux shortcuts:

• Ctrl-a s — session switcher (pick kibana-main, kibana-feat, etc.)
• Ctrl-a w — window overview within a session

To attach to a session from outside tmux: ~/dev-start.sh attach feat

Local vs Remote ES

Local ES — kbn-start.sh runs yarn es snapshot in pane 0, watches for the trigger string "succ kbn/es setup complete", then sends yarn start (with correct port arguments) to pane 1.

Remote ES — When --remote is passed, dev-start.sh generates kibana.dev.yml using credentials from ~/.kibana-remote-es.yml (fetched via oblt-cli or pasted manually). kbn-start.sh detects the remote URL and skips local ES startup. Fleet config (agent policies, Fleet Server, package installations) is also included in the remote kibana.dev.yml — the Fleet ES output host points to the remote cluster instead of host.docker.internal.

Credential renewal flow

When remote ES credentials expire or the cluster has been destroyed:

~/dev-start.sh renew              # auto-detects cluster, fetches fresh creds (all remote sessions)
~/dev-start.sh renew --session feat   # only update feat (others keep their current cluster)
~/dev-start.sh restart feat       # restart to pick up new credentials

renew auto-detects the cluster name from oblt-cli cluster list, fetches credentials via oblt-cli cluster secrets kibana-config, writes to ~/.kibana-remote-es.yml, and regenerates kibana.dev.yml for compatible remote sessions if credentials changed. Use --session <target> to limit regeneration to a single session — other sessions keep their current cluster credentials. This is useful when different sessions need to point at different clusters (e.g. testing expired certificates, schema changes, or different ES versions). Sessions whose Kibana version doesn't match the new cluster's ES version are automatically skipped.

If no cluster exists (expired and destroyed), renew offers to create a new one via oblt-cli cluster create. The create command is configurable in ~/.kibana-dev.conf via OBLT_CLUSTER_CREATE_CMD. Creation is async — run renew again after the Slack notification confirms the cluster is ready.

renew also detects version mismatches between the remote ES cluster and local Kibana (e.g. ES 9.4 vs Kibana 9.5 after a version bump on main). When a mismatch is found, it checks oblt-cli cluster list for other existing clusters and offers to switch to one if available. If no other clusters exist, it offers to destroy and replace. If some sessions still match the old ES version (e.g. a hotfix branch on 9.4), it shows which sessions are affected and adjusts options accordingly — prioritising switching to an existing cluster.

Data Ingestion

run-data must be run from within a Kibana repo directory. Auto-detect the correct directory from ~/.kibana-dev.conf (KIBANA_MAIN_DIR or WORKTREE_BASE) — don't tell the developer to navigate there manually.

run-data slo                            # Ingest SLO fake_stack data via data_forge.js
run-data synthetics                     # Create synthetics private location (Fleet Server + Agent)
run-data synthetics break <scenario>    # Trigger a Synthetics failure scenario
run-data synthetics fix <scenario>      # Restore from a failure scenario
run-data synthetics reset               # Wipe all Fleet + Synthetics state (monitors, locations, agents, agent records, policies, .fleet-* indices, API key, orphaned data)

run-data.sh reads ES host and credentials from config/kibana.dev.yml, waits for Kibana readiness via /api/status, and uses the elastic superuser (not kibana_system_user, which lacks write permissions).

Synthetics private location: Uses the Kibana synthetics_private_location.js script which starts Fleet Server, enrolls an agent, and creates the private location. Credentials are passed via --kibana-password so it works with both local and remote ES.

For remote ES, it auto-reduces concurrency (payload 1000, concurrency 1, events-per-cycle 10) to avoid timeouts.

Synthetics failure scenarios

The synthetics break and synthetics fix subcommands inject and restore Synthetics/Fleet failure states for testing diagnostic tooling. All scenarios are reversible. Monitors created by break are tagged with [BREAK] in their name so fix can clean them up.

run-data synthetics break <scenario>    # inject a failure
run-data synthetics fix <scenario>      # restore from a failure
run-data synthetics break all           # chaos mode — trigger all scenarios
run-data synthetics fix all             # full restore
run-data synthetics break help          # list available scenarios

Available scenarios:

Scenario	Break effect	Fix action
agent-offline	Stops synthetics agent Docker container	Restarts stopped agent containers
revision-mismatch	Stops agent + creates monitor (policy rev diverges)	Cleans [BREAK] monitors + restarts agent
zero-data	Creates monitor on private location with agent down	Cleans [BREAK] monitors + restarts agent
fleet-degraded	Stops Fleet Server container	Restarts Fleet Server containers
orphaned-data	Creates + deletes monitor (data remains in ES)	Deletes orphaned check data from ES
policy-disabled	Disables Fleet package policy (monitor still enabled)	Re-enables disabled package policies
orphaned-policy	Deletes monitor SO from ES (package policy remains)	Deletes package policies with no monitor
agent-unenrolled	Force-unenrolls agent (monitors still configured)	Re-enrolls agent (full synthetics setup)
service-disabled	Disables Synthetics service (API key invalidated)	Re-enables Synthetics service

Prerequisites: a working Synthetics setup with at least 1 private location, 1 enrolled agent, and 1-2 monitors. Run run-data synthetics first to provision.

Data generated by run-data slo

The slo command uses kbn-data-forge (x-pack/scripts/data_forge.js) with the fake_stack dataset and good event template. This generates data in the high-volume-metrics index with fields like http.response.status_code, service.name, etc.

Creating SLOs

Data ingestion must happen before creating SLOs — there must be data in the target index first. The standard workflow is:

1. Run run-data slo to ingest fake_stack data
2. Create an SLO via the Kibana API (see references/kibana-api.md)

When creating SLOs on data_forge data, ALWAYS use the Admin Console Availability pattern:

• Indicator type: sli.kql.custom (NOT sli.apm.transactionDuration)
• Good query: http.response.status_code < 500
• Total query: http.response.status_code : *
• Index: high-volume-metrics (NOT metrics-apm*)

This measures availability (% of non-5xx responses). This is the only SLO type that works with kbn-data-forge fake_stack data. Do NOT use APM latency indicators (sli.apm.transactionDuration) — the fake_stack dataset does not generate APM data, so APM-based SLOs will have no matching data and will fail silently.

Kibana API operations

For creating SLOs, private locations, or other resources via the Kibana API, read references/kibana-api.md for endpoint details, required headers, and example request bodies. When making API calls:

1. Read config/kibana.dev.yml to get the Kibana port and ES credentials
2. Verify Kibana is ready with ~/dev-start.sh status
3. Use the elastic superuser for authentication
4. Always include kbn-xsrf: true header

Code Quality Checks

run-checks lint       # ESLint on changed files (vs upstream/main)
run-checks typecheck  # TypeScript check on changed plugins
run-checks jest       # Unit tests on changed plugins

run-checks.sh uses git merge-base to scope checks to only the files and plugins you've actually changed on your branch. It handles both committed and untracked files.

Configuration

~/.kibana-dev.conf

User-specific paths and ports. Created by setup wizard. Uses ${VAR:-default} pattern so every value is overridable:

KIBANA_MAIN_DIR="$HOME/Documents/Development/kibana"
WORKTREE_BASE="$HOME/Documents/Development/worktrees"
ES_DATA_BASE="$HOME/Documents/Development/es_data"
MAIN_KIBANA_PORT="5602"
MAIN_ES_PORT="9201"
FEAT_KIBANA_PORT="5601"
FEAT_ES_PORT="9200"
OBLT_CLUSTER_NAME="my-cluster"   # optional, for renew auto-detect
KBN_EXTRA_ARGS="--mockIdpPlugin.enabled=false"  # optional, extra flags for yarn start

config/kibana.dev.yml

Per-worktree Kibana config. Generated from kibana.dev.yml.template (local ES) or from ~/.kibana-remote-es.yml (remote ES). Contains server port, ES hosts, and any custom Kibana settings (logging, feature flags, etc.).

Troubleshooting

For common issues and their solutions, read references/troubleshooting.md.

Key principle: always suggest dev-start.sh commands for fixes, not raw tmux commands or manual yarn start. The scripts handle port assignments and process management automatically.

Working with Remote ES (--remote)

When a session uses --remote, Kibana connects to an external Elasticsearch cluster (typically an oblt-cli-managed Cloud deployment) via cross-cluster search (CCS). This changes how indices and data are addressed.

Index patterns use a CCS prefix

Data on the remote cluster lives behind a remote_cluster: prefix. Standard local patterns do not work:

Local pattern	Remote equivalent
logs-*	remote_cluster:logs-*
metrics-apm*	remote_cluster:metrics-apm*
traces-apm*	remote_cluster:traces-apm*
filebeat-*	remote_cluster:filebeat-*

The exact prefix name (remote_cluster) varies by deployment. Check config/kibana.dev.yml — the apm_sources_access.indices and uiSettings.overrides sections show the correct prefix for the current cluster.

Discovery before creating resources (SLOs, rules, etc.)

When creating SLOs, alerting rules, or any resource that references index patterns or field values, always discover real data first:

1. Detect the CCS prefix — inspect config/kibana.dev.yml or query existing SLOs:

   curl -u "$AUTH" -H "kbn-xsrf: true" "$KIBANA_URL/api/observability/slos?perPage=5" | \
     jq '.results[].indicator.params.index'
   

2. Discover real service names — query the target index for actual service.name values:

   curl -k -u "$AUTH" "$ES_URL/<index-pattern>/_search" \
     -H "Content-Type: application/json" \
     -d '{ "size": 0, "aggs": { "services": { "terms": { "field": "service.name", "size": 30 } } } }'
   

3. Verify fields exist — confirm filters reference real fields with data:

   curl -k -u "$AUTH" "$ES_URL/<index-pattern>/_search" \
     -H "Content-Type: application/json" \
     -d '{ "size": 0, "query": { "bool": { "filter": [
       { "term": { "service.name": "<service>" } },
       { "exists": { "field": "http.response.status_code" } }
     ] } } }'
   

Never use made-up service names or assume local index patterns when connected to a remote cluster. Creating resources against non-existent data produces NO_DATA entries that clutter the UI and waste transforms.

First-time setup

~/dev-start.sh setup                    # config wizard: paths, ports, symlinks
~/dev-start.sh switch feature/my-branch # set your first feature branch
~/dev-start.sh                          # start everything