writing-live-e2e-tests

name	writing-live-e2e-tests
description	Use when writing, modifying, or running live E2E test scenarios in src/e2e/live/ for the kagura project. Triggers on Slack bot integration testing, live scenario creation, Codex/Claude provider live tests, status probe assertions, database assertions, and requests to run or debug live E2E tests.

Writing Live E2E Tests

Overview

Live E2E tests run against a real Slack workspace via Socket Mode. Each test is a standalone run-*.ts file in src/e2e/live/ that exports a LiveE2EScenario object. The CLI auto-discovers and runs them.

When the user asks to add a live E2E test, implement the scenario and run the local validation commands. When they ask to "跑下" or run it and .env.e2e exists, run the real live scenario too.

Skeleton

Every scenario file follows this structure:

import './load-e2e-env.js';
import { randomUUID } from 'node:crypto';
import fs from 'node:fs/promises';
import path from 'node:path';
import { createApplication } from '~/application.js';
import { env } from '~/env/server.js';
import type { LiveE2EScenario } from './scenario.js';
import { runDirectly } from './scenario.js';
import { SlackApiClient } from './slack-api-client.js';

interface MyResult {
  botUserId: string;
  channelId: string;
  failureMessage?: string;
  matched: {
    /* booleans for each assertion */
  };
  passed: boolean;
  rootMessageTs?: string;
  runId: string;
}

async function main(): Promise<void> {
  // 1. Guard env
  if (!env.SLACK_E2E_ENABLED) throw new Error('...');
  if (!env.SLACK_E2E_CHANNEL_ID || !env.SLACK_E2E_TRIGGER_USER_TOKEN) throw new Error('...');

  // 2. Setup
  const runId = randomUUID();
  const triggerClient = new SlackApiClient(env.SLACK_E2E_TRIGGER_USER_TOKEN);
  const botClient = new SlackApiClient(env.SLACK_BOT_TOKEN);
  const botIdentity = await botClient.authTest();

  // 3. Init result object with all matched: false
  const result: MyResult = {
    /* ... */
  };

  // 4. Start application
  const application = createApplication();
  let caughtError: unknown;

  try {
    await application.start();
    await delay(3_000);

    // 5. Post trigger message with runId marker
    // 6. Poll with deadline loop
    // 7. Assert via assertResult()
    // 8. Set result.passed = true AFTER assertion passes

    await writeResult(result);
    assertResult(result);
    result.passed = true;
    await writeResult(result);
  } catch (error) {
    result.failureMessage = error instanceof Error ? error.message : String(error);
    caughtError = error;
  } finally {
    await writeResult(result).catch(() => {});
    await application.stop().catch(() => {});
  }
  if (caughtError) throw caughtError;
}

// ALWAYS use env.SLACK_E2E_RESULT_PATH with .replace()
async function writeResult(result: MyResult): Promise<void> {
  const resultPath = env.SLACK_E2E_RESULT_PATH.replace(/result\.json$/, 'my-test-result.json');
  const absolutePath = path.resolve(process.cwd(), resultPath);
  await fs.mkdir(path.dirname(absolutePath), { recursive: true });
  await fs.writeFile(absolutePath, `${JSON.stringify(result, null, 2)}\n`, 'utf8');
}

function assertResult(result: MyResult): void {
  const failures: string[] = [];
  // Push descriptive failure strings
  if (failures.length > 0) throw new Error(`E2E failed: ${failures.join('; ')}`);
}

function delay(ms: number): Promise<void> {
  return new Promise((resolve) => {
    setTimeout(resolve, ms);
  });
}

export const scenario: LiveE2EScenario = {
  id: 'kebab-case-id',
  title: 'Human Readable Title',
  description: 'One sentence describing what is verified.',
  keywords: ['searchable', 'terms'],
  run: main,
};

runDirectly(scenario);

Key Rules

Rule	Detail
Result path	env.SLACK_E2E_RESULT_PATH.replace(/result\.json$/, 'your-name-result.json') — never hardcode
Marker pattern	Include runId in prompts and assertions: MARKER_NAME ${runId}
Polling	while (Date.now() < deadline) with delay(1_000) to delay(3_000) between iterations
Two clients	triggerClient (user token) posts prompts; botClient (bot token) polls replies
Bot identity	botClient.authTest() to get user_id for filtering replies
Application lifecycle	createApplication() → start() → delay(3_000) → test → stop() in finally
Separate helpers	Extract writeResult() and assertResult() as functions
Assert then pass	Call assertResult() first, set result.passed = true only after it succeeds
Provider-specific run	Use createApplication({ defaultProviderId: 'codex-cli' }) or 'claude-code' when the test must force a provider

Polling Pattern

const deadline = Date.now() + env.SLACK_E2E_TIMEOUT_MS;
while (Date.now() < deadline) {
  const replies = await botClient.conversationReplies({
    channel: env.SLACK_E2E_CHANNEL_ID,
    inclusive: true,
    limit: 50,
    ts: rootMessage.ts,
  });

  for (const message of replies.messages ?? []) {
    if (!message.ts || message.ts === rootMessage.ts) continue;
    if (message.user === botIdentity.user_id || message.bot_id) {
      const text = typeof message.text === 'string' ? message.text : '';
      if (text.includes(`MY_MARKER ${runId}`)) {
        result.matched.myCondition = true;
      }
    }
  }

  if (result.matched.myCondition) break;
  await delay(2_500);
}

Advanced Patterns

Anchor-then-prompt (file uploads, multi-step)

Post a non-mention anchor first, upload files or set up state, then post the mention prompt in-thread.

Multi-phase tests

Phase 1 saves state → restart application → Phase 2 verifies persistence. Each phase gets its own root message and polling loop.

Reaction lifecycle

Poll getReactions() for emoji presence/absence across phases (ack added → removed → done added).

Database validation

Use better-sqlite3 to query SQLite directly after bot processes, verify persistence of memory/session records.

Important schema details:

• memories has repo_id, not scope.
• Global memories are repo_id IS NULL.
• Workspace memories use repo_id = ?.
• Resolve DB path with path.resolve(process.cwd(), env.SESSION_DB_PATH).

Status probe

FileSlackStatusProbe writes NDJSON records. Read the probe file to verify tool progress events.

Use resetSlackStatusProbeFile(env.SLACK_E2E_STATUS_PROBE_PATH) before posting the trigger. After polling, filter records by record.threadTs === rootMessage.ts.

For Slack-visible rendering regressions, collect:

• status records: record.status and record.loadingMessages
• progress records: record.text

Assert both the positive user-facing text and absence of raw internals. Example: for Codex memory file writes, assert Saving memory... appears and /bin/zsh -lc, .kagura/runtime, and memory-ops.jsonl do not.

Validation Commands

After editing or adding a live scenario, run these from the repository root:

pnpm --filter kagura exec tsc --noEmit
pnpm exec prettier --check apps/kagura/src/e2e/live/run-your-scenario.ts

Run targeted unit tests too when the scenario covers code with existing tests, for example:

pnpm --filter kagura exec vitest run tests/codex-cli-adapter.test.ts

Check scenario discovery with required dummy env if the local shell has no live secrets loaded:

SLACK_BOT_TOKEN=xoxb-test SLACK_APP_TOKEN=xapp-test SLACK_SIGNING_SECRET=secret REPO_ROOT_DIR=/Users/innei/git/innei-repo pnpm --filter kagura exec tsx src/e2e/live/cli.ts --list your-search-term

Running Real Live E2E

Prefer running from the repository root. Because pnpm --filter kagura exec executes in the package directory, load-e2e-env.ts may not find the root .env.e2e by relative path. If .env.e2e exists at the repository root, source it explicitly:

set -a
source .env.e2e
set +a
pnpm --filter kagura exec tsx src/e2e/live/cli.ts scenario-id

If the CLI reports dotenv injected 0 variables and env validation says SLACK_BOT_TOKEN, SLACK_APP_TOKEN, SLACK_SIGNING_SECRET, or REPO_ROOT_DIR are missing, rerun with the explicit source .env.e2e command above.

Expected noisy but usually non-fatal live logs:

• Slack manifest token rotation may fail; slash command sync is skipped, but app mention E2E can still run.
• Socket Mode may log WebSocket ping/pong warnings.
• After application.stop(), Bolt may log "client has no active connection" for late events. If the scenario summary says PASS, treat this as non-blocking unless the task is specifically about shutdown behavior.

Common Mistakes

Mistake	Fix
Hardcoded result path	Use env.SLACK_E2E_RESULT_PATH.replace(...)
Missing runDirectly(scenario) at end	Required for direct tsx execution
assertResult after result.passed = true	Assert FIRST, then set passed
Forgetting application.stop() in finally	Always cleanup even on error
No runId in prompt/assertion	Every marker must include runId for test isolation
Querying memories.scope	Use repo_id IS NULL for global or repo_id = ? for workspace
Running live E2E without loaded root env	From repo root, run set -a; source .env.e2e; set +a; ...