generate-isolation-tests

name

generate-isolation-tests

description

Generates a tenant isolation test scaffold for a module's shared resource writes. Use after the comprehension gate issues REVIEW REQUIRED with an isolation evidence question on a new cache write or shared database write. Also use when adding a new shared resource write to a module that already has BEHAVIORAL_CONTRACTS.md, or when auditing an existing shared resource write that has never been tested for isolation. Reads the target module or file, identifies the resource and key structure, generates three test cases (same-tenant read, cross-tenant isolation, key collision), writes tests/isolation/<module>-isolation.test.<ext>, and updates BEHAVIORAL_CONTRACTS.md with an "Isolation guarantee" field. Invoke as: /generate-isolation-tests path/to/module /generate-isolation-tests path/to/module --interface InterfaceName /generate-isolation-tests path/to/file:line

/generate-isolation-tests

Generates tenant isolation test scaffolding for a shared resource write. The generated tests are starting points — they compile and have the right structure, but need to be adapted to your project's test framework and populated with realistic data before they pass.

Why this matters

Cross-tenant data exposure is one of the most severe dark code failure modes. A cache key like result:{query_hash} that doesn't include a tenant prefix means two tenants performing the same operation get each other's results. This is not detectable by linters, type checkers, or unit tests that run within a single tenant's scope. The only way to prove isolation is to test across tenant boundaries.

This skill generates that test. The comprehension gate can accept it as isolation evidence.

---

Arguments

• path/to/module — generates tests for all shared resource writes in the module
• path/to/module --interface InterfaceName — generates tests for a specific interface
• path/to/file:line — targets a specific cache write or DB write at a known line number

---

Phase 1: Identify the resource and its key structure

Read the specified module or file. Find all writes to shared resources:

• Redis/cache: set(key, value), setex(key, ttl, value), hset(key, field, value), framework-specific cache writes
• Database: INSERT, UPDATE, db.create(), ORM .save() on shared tables
• Message queues: publish(), emit(), send() with a topic/queue name

For each write, identify:

1. Key pattern — the structure of the cache key or the table/row identifier
2. Scope identifier — is there a user ID, tenant ID, or session ID in the key? What variable holds it?
3. Value written — what data shape goes into the resource?
4. Who reads it — grep the codebase for reads of this same key/table. Check MODULE_MANIFEST.md dependents.

If there are multiple writes, ask the user which to target if the scope is ambiguous:

I found [N] shared resource writes in this module:
1. Line [N]: cache.set(`result:${queryHash}`, ...) — no visible tenant scope
2. Line [N]: db.userSessions.create({userId, ...}) — scoped to userId
3. Line [N]: queue.publish('events', ...) — no isolation concern for queues

Which write should I generate isolation tests for? (Enter number, or "all")

---

Phase 2: Detect the isolation mechanism (or its absence)

Determine the isolation posture of the write:

Case A — Scope identifier present in key: The key includes tenant_id, user_id, org_id, or similar. The test should verify that:

• The scope identifier is always populated (not null/undefined)
• The value comes from a trusted source (auth middleware, not user input)
• Two different identifiers produce two different cache entries

Case B — No scope identifier visible: This is the risk case. The test should demonstrate what happens when two tenants perform the same operation. If the key structure genuinely doesn't need tenant scoping (e.g., it caches static data or global configuration), document that explicitly as the isolation mechanism.

Case C — Unclear / depends on runtime state: Note this in the generated test as a comment and flag it as requiring engineering judgment.

---

Phase 3: Detect the test framework

Look for: package.json (jest, vitest, mocha), pyproject.toml or pytest.ini (pytest), go.mod (testing package), Gemfile (rspec), build.gradle/pom.xml (junit).

Use the framework's native assertion style. If no framework is detectable, generate framework-agnostic pseudocode and note that it needs to be adapted.

---

Phase 4: Write the test scaffold

Write to tests/isolation/<module-name>-isolation.test.<ext>. Create the directory if it doesn't exist.

The scaffold covers three test cases. Generate each in the project's test framework:

Test 1 — Same-tenant round trip (happy path):

Given: a request authenticated as Tenant A
When: the operation is performed and the result is cached
Then: Tenant A's subsequent read returns their own data

Test 2 — Cross-tenant isolation (the critical test):

Given: two distinct tenant identities — Tenant A (id: "tenant-a") and Tenant B (id: "tenant-b")
When: Tenant A performs the operation (result cached under Tenant A's scope)
Then: Tenant B's read of the same logical key returns null, their own data, or throws
      — critically, NOT Tenant A's data

Test 3 — Key collision under identical inputs:

Given: Tenant A and Tenant B performing the same operation with identical inputs
When: Tenant A's result is cached
Then: Tenant B's cache lookup does not return Tenant A's cached value

Mark each test with a comment indicating what it verifies and what a failure means:

// TEST 2: Cross-tenant isolation
// FAILURE MEANS: A tenant can read another tenant's [data type]
// WHAT TO FIX: Add tenant_id to the cache key, or use per-tenant cache namespacing
it('does not return Tenant A data to Tenant B', async () => {
  // ... scaffold ...
  // TODO: Replace mockTenantA/mockTenantB with your project's auth context
  // TODO: Adapt cacheKey detection to your caching layer
});

At the top of the file, add:

// SCAFFOLD — This file was generated by /generate-isolation-tests.
// To activate: populate the TODOs, run [test command], verify all 3 tests pass.
// These tests will likely FAIL until populated — that is expected.
// A failing test here means the isolation mechanism is not yet verified.

---

Phase 5: Update BEHAVIORAL_CONTRACTS.md

If the target module has a BEHAVIORAL_CONTRACTS.md, append an "Isolation guarantee" field to the relevant interface section:

### Isolation guarantee

**Mechanism:** [describe what you found — e.g., "key prefixed with tenant_id from JWT claim" or
"UNVERIFIED — isolation tests generated but not yet run"]  
**Test:** `tests/isolation/<module-name>-isolation.test.<ext>`  
**Verified:** [leave blank until tests pass]  
**Generated by:** /generate-isolation-tests on [today's date]

If BEHAVIORAL_CONTRACTS.md doesn't exist, note:

No BEHAVIORAL_CONTRACTS.md found for this module.
After tests pass, run /context-layer-generator to document this module's contracts including the isolation guarantee.

---

After writing

Report:

Isolation tests generated:
✓ tests/isolation/<module-name>-isolation.test.<ext>  (3 test cases)
[✓ BEHAVIORAL_CONTRACTS.md updated — isolation guarantee section added]
[⚠ No BEHAVIORAL_CONTRACTS.md found — see note above]

Next steps:
1. Populate the TODOs in the test file with your project's tenant/auth context
2. Run: [detected test command] tests/isolation/<module-name>-isolation.test.<ext>
3. If all 3 tests pass: the isolation evidence requirement is satisfied
4. Reference this test file in the comprehension gate review as isolation evidence

If Case B (no scope identifier found), add:

⚠ No tenant scope identifier found in the cache key.
Test 2 and Test 3 are likely to fail with the current implementation.
To fix: add your tenant/user identifier to the key before the value is written.