stably-cli

name	stably-cli
preloaded	true
description	Use the Stably CLI to create, run, fix, and maintain Playwright tests in the bloom-tests repo. Use when running tests (stably test), auto-fixing failures (stably fix), or generating new tests from a prompt (stably create).

Stably CLI

Source: https://skills.sh/stablyai/agent-skills/stably-cli

AI-assisted Playwright test management: create, run, fix, and maintain tests via CLI.

Pre-flight

stably --version   # verify installed
# If not found: npm install -g stably

Requires Node.js 20+, Playwright, and a Stably account.

Command Reference

Intent	Command
Generate test from prompt	stably create "description"
Generate test from branch diff	stably create (no prompt)
Run tests	stably test
Run tests with remote env	stably --env staging test
Fix failing tests	stably fix [runId]
List recent runs	stably runs list
View run details	stably runs view <runId>
Initialize project	stably init
Install browsers	stably install [--with-deps]
List remote environments	stably env list
Inspect env variables	stably env inspect <name>
Auth	stably login / logout / whoami
Update CLI	stably upgrade [--check]

Do NOT run stably (no args) — that's an interactive AI chat for humans only.

Global Options

Option	Description
--cwd <path> / -C	Change working directory
--env <name>	Load vars from remote Stably environment
--env-file <path>	Load vars from local file (repeatable)
--verbose / -v	Verbose logging
--no-telemetry	Disable telemetry

Env var precedence (highest to lowest): Stably auth (STABLY_API_KEY) > --env > --env-file > process.env

Inspecting Runs

Use stably runs to investigate failures without browser auth (dashboard requires login we can't automate):

stably runs list                    # recent runs with status
stably runs view <runId>            # details: status, branch, commit, pass/fail/timeout counts, failed test names + errors

The run ID is the slug from the dashboard URL (e.g. ss8xdbaac1rcwm3ikcnkge0u from .../history/ss8xdbaac1rcwm3ikcnkge0u).

Core Commands

stably create [prompt...]

Generates tests from a prompt or infers from branch diff when no prompt given.

stably create "test the checkout flow"
stably create              # infer from diff
stably create "test registration" --output ./e2e

Prompt tips: be specific about user flows, UI elements, auth requirements, and error states.

stably test

Runs Playwright tests with Stably reporter. Auto-enables --trace=on. All Playwright CLI options pass through:

stably test --headed --project=chromium --workers=4
stably test --grep="login" tests/login.spec.ts
stably --env staging test --headed

stably fix [runId]

Fixes failing tests using AI analysis of traces, screenshots, logs, and DOM state.

Run ID resolution: explicit arg > CI env (GITHUB_RUN_ID) > .stably/last-run.json (24h cache). Requires git repo.

stably fix           # auto-detect last run
stably fix abc123    # explicit run ID

Typical workflow: stably test > (failures?) > stably fix > stably test

Agent fix workflow (recommended):

1. stably runs view <runId> to understand failures before fixing
2. git pull origin main to ensure you're on latest (stably fix edits files in-place)
3. stably fix <runId> with 600s+ timeout (can take 10+ min; output is mostly spinner noise)
4. Check git diff to see what stably actually changed. The terminal output is too noisy to parse
5. If main was updated since the failing run, stably's edits may conflict. Create a worktree from origin/main, manually apply the relevant changes, and PR from there
6. Review the diff critically: stably sometimes uses brittle selectors (.nth(5), .first()) or overly generic aiAssert. Prefer explicit selectors (ticker symbols, role+name) over AI-driven assertions when possible

Long-Running Commands (Important for Agents)

stably create and stably fix are AI-powered and can take several minutes.

• Use exec with a long timeout (600s+) or background mode
• All other commands complete in seconds

Bloom-Specific Setup

Prefer the repo-local CLI if stably is not globally installed. Install dependencies first, then call ./node_modules/.bin/stably directly so verification does not depend on global npm state.

cd ~/projects/bloom-tests
npm install --no-audit --no-fund
source ~/.hermes/.env
STABLY_API_KEY=$STABLY_API_KEY \
STABLY_PROJECT_ID=$STABLY_PROJECT_ID \
BASE_URL=https://bloom.onrender.com \
./node_modules/.bin/stably test

To fix after a failing run:

source ~/.hermes/.env
STABLY_API_KEY=$STABLY_API_KEY STABLY_PROJECT_ID=$STABLY_PROJECT_ID ./node_modules/.bin/stably fix

If a test run URL is produced but stably runs view fails because stably is not on PATH, use the local binary:

source ~/.hermes/.env
STABLY_API_KEY=$STABLY_API_KEY STABLY_PROJECT_ID=$STABLY_PROJECT_ID \
./node_modules/.bin/stably runs view <runId>

Playwright Pitfalls

isVisible({ timeout }) does NOT wait. Playwright's locator.isVisible() returns immediately regardless of any timeout option passed. The timeout is silently ignored. To wait for an optional element, use locator.waitFor({ state: 'visible', timeout: 3000 }) inside a try/catch. This is the #1 cause of race conditions in modal dismissal guards.

// WRONG — returns immediately, modal may not have rendered yet
if (await modal.isVisible({ timeout: 3000 })) { ... }

// RIGHT — actually waits up to 3s for the element
try {
  await modal.waitFor({ state: 'visible', timeout: 3000 });
  await page.keyboard.press('Escape');
  try {
    await modal.waitFor({ state: 'hidden', timeout: 1000 });
  } catch {
    await modal.locator('..').locator('button').first().click({ force: true });
  }
} catch {
  // Modal never appeared — nothing to do
}

Strict mode violations with getByRole('button'). If a row or container has multiple buttons, getByRole('button') without .first() or a name filter throws a strict mode error. Always scope: .getByRole('button').first() or .getByRole('button', { name: 'Bookmark' }).

afterAll cleanup is mandatory for state-mutating tests. Tests that bookmark stocks, create portfolios, or modify user state MUST have an afterAll hook to undo those changes. Watchlist/bookmark state persists across runs and causes cascading failures in other tests. Pattern:

test.afterAll(async ({ browser }) => {
  const page = await browser.newPage();
  try {
    // Navigate and undo state changes
  } catch {
    // Cleanup target doesn't exist — nothing to do
  } finally {
    await page.close();
  }
});

Bloom Test Failure Patterns

Common failure modes in bloom-tests and fixes:

Feedback Modal Overlay: A "Bloom experience" feedback modal overlays form elements (portfolio creation, edit flows), causing timeouts. Dismiss early using the waitFor pattern above with page.getByRole('heading', { name: /Bloom experience/i }).

Company Name vs Ticker Selectors: Bloom UI renders ticker symbols (GOOGL, MSFT) as link text, not company names. Use { name: /GOOGL/ } not { name: 'Alphabet' }.

agent.act() Failures: Replace with explicit Playwright selectors when possible. Reserve agent.act() for genuinely dynamic interactions where the exact selector isn't predictable.

Search/Algolia Flakiness: Avoid making critical Bloom QA flows depend on live search results when the same state can be reached through deterministic in-app navigation. In Stably/headless runs, stock search for AAPL has returned ETF-like results (AAPW, AAPY, AAPD) or hung before Apple Inc appeared. Prefer direct collection pages, ticker rows, or “Copy collection to portfolio” flows when testing portfolio mechanics. See references/bloom-search-and-collections-flakiness.md for a concrete repro and refactor pattern.

Back Button Fragility: .getByRole('button').first() is brittle. Use name or aria-label selectors instead.

Required Env Vars

STABLY_API_KEY=<your-stably-api-key>
STABLY_PROJECT_ID=<your-stably-project-id>

Playwright Config (Reporter Setup)

import { defineConfig } from '@playwright/test';
import { stablyReporter } from '@stablyai/playwright-test';

export default defineConfig({
  use: { trace: 'on' },
  reporter: [
    ['list'],
    stablyReporter({
      apiKey: process.env.STABLY_API_KEY,
      projectId: process.env.STABLY_PROJECT_ID,
    }),
  ],
});

Troubleshooting

Problem	Solution
"Not authenticated"	stably login
API key not recognized	stably whoami to verify
Tests in wrong directory	stably create "..." --output ./tests/e2e
Missing browser	stably install --with-deps
Traces not uploading	Set trace: 'on' in playwright.config.ts
"Run ID not found"	Run stably test first, then stably fix
Dashboard URL but no browser auth	Use stably runs view <runId> via CLI instead

Links

• Docs: https://docs.stably.ai
• CLI Quickstart: https://docs.stably.ai/stably2/cli-quickstart
• Dashboard: https://app.stably.ai
• API Keys: https://auth.stably.ai/org/api_keys/