troubleshooting

name

troubleshooting

description

Use for multi-layer fault isolation when behavior differs across contexts such as terminal vs desktop, interactive shell vs service, local CLI vs plugin, one OS vs another, or when clean logs do not prove the feature works. Guides agents to converge by proving one layer at a time, preserving evidence, and changing only the layer that remains unproven. Skip for obvious single-layer mistakes such as syntax errors or known one-line typos.

Troubleshooting: Layer Convergence Framework

Overview

Most hard bugs are not hard because the root cause is obscure. They are hard because the system has many layers, and each layer can fail independently.

The natural instinct is to enumerate possible causes and test them one by one. That's O(n) - and when you don't know the full list of possibilities, it's O(unknown), which is unbounded.

This framework inverts the approach: instead of looking for the bug, prove each layer is good. The first layer you cannot prove becomes the next evidence boundary.

Trigger / When to Use

Load this skill when any of these are true:

1. The failure is context-dependent: works in a shell but not a desktop app, works interactively but not from a service, works in a CLI but not a plugin, or works on one OS/runtime but not another.
2. The system crosses multiple boundaries: host process, plugin, hook, CLI binary, shell, API, config loader, OS service, network layer, or datastore.
3. Logs are clean but behavior is still wrong.
4. You are tempted to guess the root cause before proving where the signal disappears.

Do NOT load for single-layer issues: syntax error, missing file, wrong URL, known config typo, or another already-localized mistake. Those are faster to fix directly.

Core Methodology: Layer Convergence

The Mindset

Don't find the bug. Prove every layer is good. The first layer you cannot prove is where you instrument next.

Each layer in a system has its own truth. A statement that is true at layer N does not imply it is true at layer N+1.

Common false equivalence	Why it fails
"Shell can find foo" → "Desktop can find foo"	Different PATH, different init scripts
"Logs are clean" → "Feature works"	Clean init ≠ correct runtime
"CLI works" → "Plugin integration works"	Different calling convention, different argv
"I fixed a warning" → "I fixed the bug"	Warnings are noise, not root cause

The 7-Layer Stack

For a typical plugin/bridge/integration problem, verify in this order. Each layer, once confirmed, eliminates an entire class of hypotheses.

Layer 1: External tool exists and works in isolation
Layer 2: Host process can find the external tool
Layer 3: Plugin/bridge code is loaded and active
Layer 4: Hook/callback is actually triggered
Layer 5: Data structures at the hook point match expectations
Layer 6: The integration call (CLI invocation, API call) is correct
Layer 7: The result is correctly written back

Do not skip layers without writing down why. If a layer cannot be proven, do not immediately declare it the root cause; turn it into a smaller probe.

Workflow

Step 0: Classify the logs

Before touching anything, read all available logs and sort each line into one of three buckets:

• Noise warnings: runtime warnings, deprecation notices, module type hints. Fix them only if they obscure other signals.
• Loading/init errors: module not found, config parse failure. These indicate the system didn't start correctly.
• Active failure signals: explicit "disabled", "failed", "threw". These are first priority.

The goal is not to make logs clean. The goal is to identify the strongest signal, then work from there.

Step 1: Prove the external tool works (Layer 1)

Before debugging any integration, prove the dependency is sound in isolation:

# Prefer a resolved binary path over whatever the current shell happens to find.
tool_path="$(command -v tool)"
"$tool_path" --version
"$tool_path" subcommand 'test input'

Verify with at least three representative inputs:

• A simple case that should always work
• A realistic case that matches the failing scenario
• A case that should produce no output (null/empty)

If the tool itself fails here, the problem is upstream — no point debugging the integration.

Step 2: Prove the host process can see the tool (Layer 2)

Desktop apps, services, and daemons often have a shorter PATH than your interactive shell. Do not assume equivalence.

Strategies:

• Check process.env.PATH (Node), System.getenv("PATH") (Java), etc.
• Add platform-appropriate fallback directories only as examples, not as the skill's source of truth.
• Resolve the binary at init time using an executable check such as accessSync(path, X_OK).
• Store the resolved absolute path, do not re-resolve at runtime

Step 3: Prove the bridge loads (Layer 3)

If the bridge code (plugin, extension, middleware) didn't load, nothing downstream matters.

Verification:

• Look for init-phase log output (explicit "loaded" / "active" / "enabled")
• If absent, add a single startup log line: [plugin] active
• If the plugin has a disabled/fallback path, check which path was taken
• Fix loading issues first (wrong install path, missing dependency, module type mismatch) before looking at runtime behavior

Step 4: Prove the hook fires (Layer 4)

If the hook/callback/interceptor doesn't trigger, the integration is dead.

Add temporary instrumentation (one line per question):

console.log(`[plugin] tool.execute.before triggered`)  // Did it fire?
console.log(`[plugin] tool=${(input as any).tool}`)     // What tool?
console.log(`[plugin] argKeys=${Object.keys(args)}`)    // What args exist?
console.log(`[plugin] command type=${typeof args.command}`) // Is command a string?

Each log line must answer exactly one yes/no question. Do NOT dump full objects — that's noise, not signal.

Step 5: Prove hook data structures match expectations (Layer 5)

Once the hook fires, prove that the data you plan to mutate is the data the downstream layer actually reads.

Verify:

• The expected field exists.
• The field has the expected type.
• The field contains the original input, not a normalized or already-mutated representation.
• You can identify the exact object that must be updated.

Do not proceed to the integration call until this is proven. Otherwise a successful call can still appear to fail because the result is written to the wrong object.

Step 6: Prove the call is correct (Layer 6)

This is the most common failure point. The key insight:

If you pass a string containing spaces, pipes, quotes, or globs to a shell, the shell will reinterpret it. You lose argv boundaries.

Wrong (shell template string):

await $`tool subcommand ${userInput}`.quiet().nothrow()

Correct (argv array):

const result = spawnSync(toolBinary, ["subcommand", userInput], {
  encoding: "utf8",
})

The difference: the first passes a string to a shell (which re-tokenizes it); the second passes a typed argv array to the OS (which preserves boundaries).

Step 7: Prove writeback is correct (Layer 7)

If the call succeeded but the result was not written back to the right place, the integration still fails.

Verify:

• The result is only written back when non-null / non-empty
• The writeback targets the same data structure the downstream reads from
• A failed rewrite leaves the original value unchanged (safe degradation)

Step 8: Remove probes, verify end-to-end

Delete all temporary instrumentation. Verify the fix with a clean run.

Evidence Template

When reporting findings or deciding what to change, keep the proof compact:

Layer:
Question:
Probe:
Evidence:
Verdict:
Next step:

Use this format until the failing boundary is localized. After the fix, report:

Changed:
Why this layer:
Verification:
Residual risk:

Common Anti-Patterns

Anti-pattern	Why it's wrong
Changing PATH, hook, and rewrite logic at once	If it works, you don't know which fix mattered
"Let me add logging everywhere"	Noise increases, signal doesn't
Fixing warnings and declaring victory	Clean logs ≠ working feature
Copying the tool's rules into the plugin	Duplicates responsibility, diverges over time
Blaming the upstream tool without verifying in isolation	Wastes time if the tool is fine

Guardrails

• One variable at a time. Change one thing, test, then move on.
• External before internal. Prove the tool works outside the integration, then prove the integration sees the tool, then debug the integration logic.
• The bridge does NOT reimplement the tool. A plugin should:
  1. Find the binary (resolve)
  2. Call it with correct argv (spawn)
  3. Return the result (writeback) Nothing else. No shell parsing, no rule replication, no AST guessing.
• Probes are deleted after use. Temporary instrumentation is a tool, not a feature. Remove it before declaring done.
• If you can't prove a layer, you haven't ruled it out. Don't assume a layer is fine because "it should be" — prove it.

Reference

For a concrete example, read references/opencode-rtk-case-study.md only when you need a full worked case. The reference is bundled with this skill so the skill does not depend on user-specific absolute paths.