# oblt-playwright
A Playwright framework for functional validation and end-user performance assessment of the Elastic Observability solution across deployment types, targeting the latest released version.
The framework provides three complementary ways to run Observability checks:
- Product journeys under `tests/kibana/product-journeys` validate end-to-end user workflows and product behavior.
- Walkthrough suites under `tests/kibana/perf-walkthroughs` cover fixed UI walkthroughs across key pages, measuring load/render performance against an expected data volume.
- Benchmark scenario runners under `tests/bb` generate tests from external scenario definitions, so the same runner can measure different URLs, data views, host limits, zoom levels, and time ranges without changing test code.
Benchmark runners also collect browser performance metrics and network traces. Reports include Web Vitals, navigation timing, Chrome runtime work metrics, request summaries, and the slowest API/static requests for each measured pass.
## 📦 Getting Started
### Prerequisites
#### Datasets
Test suites ([APM](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-apm.md), [Dataset Quality](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-datasets.md), [Hosts](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-hosts.md), [Infrastructure](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-infra.md), [Logs](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-logs.md)) require a deployment/project to ingest scenario-specific data, examine [required datasets](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/data-mapping.md) accordingly.
Note: APM scenarios are strictly bound to data generated by [apmsoak](https://github.com/elastic/apm-perf/tree/main/cmd/apmsoak).
Product journeys are bound to data generated by `superdemo`—a set of instrumented microservices that continuously generate realistic traffic.
#### Background tasks
To assess the performance impact of alerting rules, you can create them before running tests. Use the following command to establish a set of rules that are checked every minute:
```bash
npx playwright test alerting-rules.api.spec.ts --project api
```
This will create alerting rules for the following rule types:
- `.es-query`
- `apm.anomaly`
- `apm.error_rate`
- `apm.transaction_duration`
- `apm.transaction_error_rate`
- `metrics.alert.inventory.threshold`
## 🤖 CI
### 🔒 Environmental variables
#### GitHub Actions Secrets:
```typescript
API_KEY_SERVERLESS =
API_KEY_STATEFUL =
ELASTICSEARCH_HOST_SERVERLESS =
ELASTICSEARCH_HOST_STATEFUL =
KIBANA_HOST_SERVERLESS =
KIBANA_HOST_STATEFUL =
KIBANA_PASSWORD_SERVERLESS =
KIBANA_PASSWORD_STATEFUL =
KIBANA_USERNAME_SERVERLESS =
KIBANA_USERNAME_STATEFUL =
REPORT_CLUSTER_ES =
REPORT_CLUSTER_API_KEY =
```
#### GitHub Actions Variables:
```typescript
START_DATE = '2025-06-24T00:00:00.000Z' // Set when the absolute time range is used
END_DATE = '2025-06-25T00:00:00.000Z' // Set when the absolute time range is used
BB_SCENARIOS_FILE = './path/to/scenarios.json' // Set when running benchmark scenario runners
// Predefined variables:
TIME_UNIT_HOURS
TIME_UNIT_DAYS
TIME_VALUE_1
TIME_VALUE_3
TIME_VALUE_7
TIME_VALUE_14
TIME_VALUE_29
```
### 🚀 Workflows
Currently configured workflows (see [ECH](https://github.com/elastic/oblt-playwright/blob/main/.github/workflows/ech.yml) | [Serverless ](https://github.com/elastic/oblt-playwright/blob/main/.github/workflows/serverless.yml)) execute each of the test suites ([APM](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-apm.md), [Dataset Quality](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-datasets.md), [Hosts](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-hosts.md), [Infrastructure](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-infra.md), [Logs](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-logs.md)) on an hourly basis with 15 minute intervals between them.
- APM test suite executes at the top of each hour.
- Dataset Quality executes 15 minutes past the hour.
- Infrastructure test suite executes 30 minutes past the hour.
- Logs test suite executes 45 minutes past the hour.
Tests are executed sequentially, with only one Kibana user at a given time.
#### To use the absolute time range:
1. Set `START_DATE` and `END_DATE` GitHub Actions variables.
2. Set `ABSOLUTE_TIME_RANGE` workflow job environmental variables to `true`.
### 📊 Test reports
A custom reporter generates JSON files and sends them to a remote Elasticsearch reporting cluster. This is configured via the `REPORT_CLUSTER_ES` and `REPORT_CLUSTER_API_KEY` environmental variables.
Benchmark scenario runners extend the main JSON report with performance metrics for each measured navigation. When a runner collects traces, it also writes an additional `*.network-trace.json` file containing the request summary and slowest request breakdown for the same measured navigation.
Product journeys generate a test-level JSON report containing the result status, duration, and errors. They do not collect walkthrough step timings or benchmark performance metrics.
Journey reports also carry `cluster_name`, derived from the `KIBANA_HOST` hostname, so an alert can name the cluster that failed. When a journey fails, the screenshot is uploaded to the reporting cluster's Kibana and the report gets a `screenshot_url` pointing at it.
## 💻 Local usage
### 🛠️ Installation
Make sure you have [Node.js](https://nodejs.org/en/download) installed.
1. Clone the repository.
```bash
git clone https://github.com/elastic/oblt-playwright.git
```
2. Install dependencies:
```bash
npm install
```
3. Install Chromium:
```bash
npx playwright install chromium
```
### ⚙️ Setup
#### 🔒 Environmental variables
```typescript
KIBANA_HOST =
ELASTICSEARCH_HOST =
KIBANA_USERNAME =
KIBANA_PASSWORD =
ELASTICSEARCH_USER = // Optional; HTTP Basic auth username
ELASTICSEARCH_PASSWORD = // Optional; HTTP Basic auth password
TIME_UNIT = 'Minutes'
TIME_VALUE = 15
ABSOLUTE_TIME_RANGE = false // Set 'true' to use the absolute time range
START_DATE = '2025-06-24T00:00:00.000Z' // Set when the absolute time range is used
END_DATE = '2025-06-25T00:00:00.000Z' // Set when the absolute time range is used
API_KEY =
BB_SCENARIOS_FILE = './path/to/scenarios.json' // Set when running benchmark scenario runners
REPORT_DIR = './playwright-report' // Optional; local JSON report directory
```
Set `ELASTICSEARCH_USER` and `ELASTICSEARCH_PASSWORD` together only when the project requires HTTP Basic authentication.
#### 📂 Test report location
Reports are written to `playwright-report` by default. The directory is created automatically; set `REPORT_DIR` to use a different location.
### ▶️ Running tests
By default, tests run in headless mode. To launch browsers in headed mode, use the `--headed` flag.
It is recommended to run tests during ongoing data collection.
#### Run all performance walkthroughs
```bash
npx playwright test --project walkthrough --headed
```
#### Run a specific performance walkthrough
```bash
npx playwright test apm.walkthrough.spec.ts --project walkthrough --headed
```
#### Run a specific performance test
```bash
npx playwright test -g "APM - Services" --project walkthrough --headed
```
#### Run product journeys
Before running a journey, ensure that the target deployment is ingesting the required `superdemo` data.
Run all product journeys:
```bash
npx playwright test tests/kibana/product-journeys --project journey --headed
```
Run the Superdemo product journey:
```bash
npx playwright test tests/kibana/product-journeys/superdemo.journey.spec.ts --project journey --headed
```
#### Run benchmark scenarios
Benchmark runners read scenario parameters from `BB_SCENARIOS_FILE`. Each scenario is filtered by its `section`, for example `hosts` scenarios are executed by `hosts.bb.spec.ts` and `discover` scenarios are executed by `discover.bb.spec.ts`.
```json
[
{
"name": "Hosts 500",
"section": "hosts",
"host_limit": 500,
"zoom": 0.9
}
]
```
```bash
BB_SCENARIOS_FILE=./scenarios.json npx playwright test tests/bb/hosts.bb.spec.ts --project bb --headed
```
The Hosts benchmark runner performs a setup navigation to discover Kibana's canonical URL state, then runs three measured navigations for each scenario: `cold`, `warm1`, and `warm2`. Each measured pass produces its own JSON report and network trace report, making cold-cache and warmed steady-state results directly comparable.
## 📚 Additional Resources
- [Test suites and required datasets](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/data-mapping.md)
- Step definitions: [APM](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-apm.md) | [Dataset Quality](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-datasets.md) | [Hosts](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-hosts.md) | [Infrastructure](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-infra.md) | [Logs](https://github.com/elastic/oblt-playwright/blob/main/docs/perf-walkthroughs/walkthrough-logs.md)
- Benchmark scenario runners: [Hosts overview](https://github.com/elastic/oblt-playwright/blob/main/docs/bb-scenario-runners/hosts-benchmark-overview.md)