learning-sample

name	learning-sample
description	End-to-end multilingual demo: generate English docs, ingest via CDP, ask questions in 13 languages via Playwright, verify TerranSoul replies in the matching language. Spec-driven loop until all pass.
argument-hint	[optional: path to an existing ZIP, or leave empty to generate sample docs]
compatibility	Requires TerranSoul running with CDP on :9222 (npm run dev:desktop-e2e) and Ollama
user-invocable	true
disable-model-invocation	false

What this skill tests

TerranSoul learns 13 English documents (one per landmark). Then a Playwright test sends questions in 13 different languages via CDP and asserts that TerranSoul replies in the same language it was asked — not always English.

This proves:

1. Brain retrieval works from English sources
2. The LLM understands questions in 13 languages
3. The LLM replies in the language it was addressed in

---

Step 0 — Verify prerequisites

# TerranSoul must be running with CDP exposed:
npm run dev:desktop-e2e   # starts Tauri + Vite with --remote-debugging-port=9222

# Verify CDP is reachable:
curl -s http://127.0.0.1:9222/json/version | head -3

If the app is not running, print:

"Please run npm run dev:desktop-e2e in a separate terminal, then retry." Wait for the user to confirm the app is running.

---

Step 1 — Generate the English sample ZIP

If no ZIP path was given as $ARGUMENTS:

Create tmp/learning-sample-<YYYYMMDDTHHMMSS>/ and write these 13 English plain-text files. Each file has 3–4 factual sentences and one KEY FACT: line.

Filename	Topic	Key fact
hoian.txt	Hoi An Ancient Town, Vietnam	Located in Quang Nam Province
fuji.txt	Mount Fuji, Japan	Located in Shizuoka and Yamanashi
greatwall.txt	Great Wall of China	Mainly built during the Ming Dynasty
gyeongbok.txt	Gyeongbokgung Palace, South Korea	Built in 1395, Joseon Dynasty
petra.txt	Petra, Jordan	Located in southern Jordan
sagrada.txt	Sagrada Família, Barcelona, Spain	Designed by Antoni Gaudí
versailles.txt	Palace of Versailles, France	Located in Versailles, Île-de-France
neuschwanstein.txt	Neuschwanstein Castle, Germany	Located in Bavaria (Bayern)
redeemer.txt	Christ the Redeemer, Brazil	Located in Rio de Janeiro
tajmahal.txt	Taj Mahal, India	Located in Agra, Uttar Pradesh
baikal.txt	Lake Baikal, Russia	Located in Siberia, Irkutsk Oblast
doiinthanon.txt	Doi Inthanon National Park, Thailand	Located in Chiang Mai Province
borobudur.txt	Borobudur Temple, Indonesia	Located in Central Java Province

Example file (hoian.txt):

Hoi An Ancient Town is a well-preserved trading port in central Vietnam,
recognized as a UNESCO World Heritage Site in 1999. It is famous for its
lantern-lit streets, wooden merchant houses, and blend of Vietnamese,
Chinese, and Japanese architectural styles.

KEY FACT: Hoi An Ancient Town is located in Quang Nam Province, Vietnam.

Then ZIP all files:

cd tmp/learning-sample-<ts>
zip learning-sample.zip *.txt

Print the absolute path: <repo>/tmp/learning-sample-<ts>/learning-sample.zip

---

Step 2 — Write the spec (spec-kit)

Call /speckit-specify with:

"Multi-language reply verification via CDP automation: TerranSoul ingests 13 English landmark documents, then a Playwright test sends questions in 13 different languages via CDP and verifies each reply is in the same language as the question, not English."

Then call /speckit-plan and /speckit-tasks. Record SPEC_DIR.

---

Step 3 — Generate the Playwright spec

Create Real-E2E/learning-sample/learning-sample.spec.ts with this structure:

/**
 * Real-E2E: Multilingual reply verification.
 * Spec: SPEC_DIR/spec.md
 *
 * Prerequisites:
 *   1. npm run dev:desktop-e2e   (Tauri CDP on :9222)
 *   2. Ollama running with a multilingual model (gemma4:12b recommended)
 *   3. ZIP ingested (see test.beforeAll)
 */
import { test, expect } from '@playwright/test';
import { existsSync } from 'node:fs';
import { join } from 'node:path';
import {
  connectToDesktopApp,
  sendMessage,
  setScenarioContext,
  waitForThinkingDone,
} from '../_shared/helpers';

const ZIP_PATH = join(
  __dirname, '..', '..', 'tmp', 'learning-sample-<ts>', 'learning-sample.zip'
);

/** Probe questions — written entirely in the target language. */
const PROBES = [
  { code: 'vi', q: 'Hội An cổ kính nằm ở tỉnh nào của Việt Nam?',                         topic: 'Quang Nam' },
  { code: 'ja', q: '富士山は日本のどの県にまたがっていますか？',                             topic: '静岡・山梨' },
  { code: 'zh', q: '长城主要是在哪个朝代修建的？',                                           topic: '明朝' },
  { code: 'ko', q: '경복궁은 어느 왕조 시대에 건립되었나요？',                              topic: '조선' },
  { code: 'ar', q: 'في أي دولة تقع مدينة البتراء الأثرية؟',                                 topic: 'الأردن' },
  { code: 'es', q: '¿Quién diseñó la Sagrada Família en Barcelona?',                        topic: 'Gaudí' },
  { code: 'fr', q: 'Où se trouve le Château de Versailles en France ?',                     topic: 'Versailles' },
  { code: 'de', q: 'In welchem deutschen Bundesland befindet sich Schloss Neuschwanstein?', topic: 'Bayern' },
  { code: 'pt', q: 'Em qual cidade brasileira está localizado o Cristo Redentor?',          topic: 'Rio' },
  { code: 'hi', q: 'ताज महल भारत के किस शहर में स्थित है?',                               topic: 'आगरा' },
  { code: 'ru', q: 'В каком регионе России находится озеро Байкал?',                        topic: 'Сибирь' },
  { code: 'th', q: 'ดอยอินทนนท์ตั้งอยู่ในจังหวัดใดของประเทศไทย?',                          topic: 'เชียงใหม่' },
  { code: 'id', q: 'Candi Borobudur terletak di provinsi apa di Indonesia?',                topic: 'Jawa Tengah' },
];

/** Detect dominant language from reply text using Unicode ranges + stop-words. */
function detectLanguage(text: string): string {
  if (/[぀-ヿ]/.test(text)) return 'ja';           // Hiragana/Katakana
  if (/[가-힯]/.test(text)) return 'ko';           // Hangul
  if (/[一-鿿]/.test(text)) return 'zh';           // CJK
  if (/[؀-ۿ]/.test(text)) return 'ar';           // Arabic
  if (/[ऀ-ॿ]/.test(text)) return 'hi';           // Devanagari
  if (/[Ѐ-ӿ]/.test(text)) return 'ru';           // Cyrillic
  if (/[฀-๿]/.test(text)) return 'th';           // Thai
  // Latin-script stop-words
  if (/\b(của|tỉnh|nằm ở|là)\b/.test(text)) return 'vi';
  if (/\b(está|ciudad|en qué|diseñó)\b/.test(text)) return 'es';
  if (/\b(se trouve|est|dans|où)\b/.test(text)) return 'fr';
  if (/\b(befindet|liegt|Bundesland|welchem)\b/.test(text)) return 'de';
  if (/\b(está|localizado|cidade|qual)\b/.test(text)) return 'pt';
  if (/\b(terletak|provinsi|adalah|di)\b/.test(text)) return 'id';
  return 'en';
}

async function lastAssistantMessage(page: any, sinceIndex: number): Promise<string> {
  return page.evaluate((idx: number) => {
    const app = (document.querySelector('#app') as any)?.__vue_app__;
    const messages = app?.config?.globalProperties?.$pinia?._s?.get('conversation')?.messages ?? [];
    for (let i = messages.length - 1; i >= idx; i--) {
      const m = messages[i];
      if (m.role === 'assistant' && typeof m.content === 'string' && m.content.length > 0)
        return m.content as string;
    }
    return '';
  }, sinceIndex);
}

async function messageCount(page: any): Promise<number> {
  return page.evaluate(() => {
    const app = (document.querySelector('#app') as any)?.__vue_app__;
    return app?.config?.globalProperties?.$pinia?._s?.get('conversation')?.messages?.length ?? 0;
  });
}

test.describe('Real-E2E: Multilingual reply verification (13 languages)', () => {
  test.beforeEach(async (_p, testInfo) => setScenarioContext(testInfo));

  test.beforeAll(async () => {
    // Verify ZIP exists (generated in Step 1)
    expect(existsSync(ZIP_PATH), `ZIP not found: ${ZIP_PATH}. Run /learning-sample first.`).toBe(true);
  });

  test('ingest English docs then reply in 13 languages', async (_p, testInfo) => {
    test.setTimeout(20 * 60 * 1000); // 20 min

    const { browser, page } = await connectToDesktopApp();

    // ── Ingest ZIP ───────────────────────────────────────────────────────
    const beforeIngest = await messageCount(page);
    await sendMessage(page, `/ingest ${ZIP_PATH}`);
    // Wait for indexing confirmation (up to 3 min for 13 files)
    await expect(async () => {
      const reply = await lastAssistantMessage(page, beforeIngest);
      expect(reply).toMatch(/indexed|learned|ingested|\d+\s+doc/i);
    }).toPass({ timeout: 3 * 60 * 1000, intervals: [5_000] });

    // ── Probe all 13 languages ───────────────────────────────────────────
    const results: { code: string; detectedLang: string; reply: string; pass: boolean }[] = [];

    for (const probe of PROBES) {
      const idx = await messageCount(page);
      await sendMessage(page, probe.q);

      // Wait for a non-empty assistant reply
      let reply = '';
      await expect(async () => {
        reply = await lastAssistantMessage(page, idx);
        expect(reply.length).toBeGreaterThan(10);
      }).toPass({ timeout: 60_000, intervals: [2_000] });

      const detectedLang = detectLanguage(reply);
      const pass = detectedLang === probe.code;
      results.push({ code: probe.code, detectedLang, reply: reply.slice(0, 120), pass });

      // Short pause between probes to avoid flooding
      await page.waitForTimeout(1_500);
    }

    // ── Print result table ───────────────────────────────────────────────
    console.log('\n=== Multilingual Reply Results ===');
    console.log('Code | Detected | Pass | Reply (first 120 chars)');
    console.log('-----|----------|------|------------------------');
    for (const r of results) {
      console.log(`${r.code.padEnd(4)} | ${r.detectedLang.padEnd(8)} | ${r.pass ? '✅' : '❌'}   | ${r.reply}`);
    }

    const failed = results.filter(r => !r.pass);
    if (failed.length > 0) {
      console.warn('\nFailed languages:', failed.map(f => f.code).join(', '));
    }

    // ── Assert all pass ──────────────────────────────────────────────────
    expect(
      failed.map(f => `${f.code}: got ${f.detectedLang}`),
      'Some languages replied in wrong language. Add reasoning rule: "Reply in same language as question."'
    ).toHaveLength(0);

    await browser.close();
  });
});

Replace <ts> in ZIP_PATH with the actual timestamp from Step 1.

---

Step 4 — Run the spec

npx playwright test \
  --config Real-E2E/playwright.config.ts \
  Real-E2E/learning-sample/learning-sample.spec.ts \
  --reporter=line

Stream the output with:

# In a separate terminal:
Get-Content Real-E2E/learning-sample/output/latest.log -Wait

---

Step 5 — Loop: fix failures

For each failing language code in the test output:

Fix A — Wrong language reply (knows answer but replies in English) Add a reasoning rule via Brain settings or invoke:

await invoke('add_reasoning_rule', {
  name: 'Match reply language to question language',
  rule: 'Always reply in the same language the user wrote their message in. ' +
        'If the question is in French, reply in French. ' +
        'If the question is in Japanese, reply in Japanese.',
  enabled: true,
});

Then re-run the test.

Fix B — Wrong or no answer (document not retrieved) Re-ingest the specific file with tags:

/ingest <path>/<lang>.txt --tags "landmark,location,<country-name>"

Then re-run.

Fix C — Model too small (< 3B params can't handle script) Switch to gemma4:12b via the model switcher pill in the TerranSoul chat header. Then re-run.

Re-run up to 3 times per language. After 3 attempts, document as known limitation with model recommendation.

---

Step 6 — Report and commit spec artefacts

After all languages pass (or after 3 attempts):

1. Write SPEC_DIR/test-results.md with the final pass/fail table.
2. Mark tasks complete in SPEC_DIR/tasks.md.
3. Add milestone to rules/milestones.md:

   | REAL-E2E-SPEC-<NNN> | Multilingual reply verification (13 languages) | done | SPEC_DIR |
   

4. Sync to brain — in TerranSoul chat type /reflect to persist the outcome.

---

Done When

• npm run dev:desktop-e2e running (CDP on :9222 confirmed)
• English ZIP at tmp/learning-sample-<ts>/learning-sample.zip (13 files)
• Spec written at SPEC_DIR/spec.md via /speckit-specify
• Real-E2E/learning-sample/learning-sample.spec.ts written
• npx playwright test runs green — all 13 languages ✅
• Failures fixed and re-run (or documented as known limitations)
• SPEC_DIR/test-results.md written
• Milestone row added to rules/milestones.md
• /reflect typed in TerranSoul chat