creating-debug-tests-and-iterating

name	creating-debug-tests-and-iterating
description	Use this skill when faced with a difficult debugging task where you need to replicate some bug or behavior in order to see what is going wrong.

*CRITICAL* Add the following steps to your Todo list using TodoWrite:

From this point on, ignore any existing tests until you have a working example validated through a new test file.

1. Write a script that interacts with the application from the outside. The script should not call any internals. It should only interact with the external interfaces.
2. Check to see if you require authentication. If you do, ask me for credentials. Do NOT use mock mode or test harnesses. You should be testing the real thing.
3. Follow these steps in a loop until the bug is fixed:

• Add many logs to the application. You MUST do this on every loop.
• Run the debug script.
• Analyze the output: read logs, identify errors, do whatever you need to.
• Update the debug script. If you get stuck: did you add logs?

5. Identify and fix the issue at hand.
6. Clean up all background jobs, and remove extraneous logs.
7. Make sure other tests pass.

Debug Testing

To test different kinds of applications, write scripts that test the application interfaces. Your testing should be as close to 'real' as possible.

Example

Identify the application boundary to be tested and the tools you need to test it.

CLI Tool:

./path/to/cli.sh arg1 arg2

subprocess.run(["./path/to/cli.sh", "arg1", "arg2"])

exec('./path/to/cli.sh', (error, stdout, stderr) => {
  if (error) {
    console.error(`exec error: ${error}`);
    return;
  }
  console.log(`stdout: ${stdout}`);
  console.error(`stderr: ${stderr}`);
});

API:

Start the server:

cd backend && python server.py&
cd frontend && npm run dev&

Call to the server using scripting language of choice.

Do NOT get in a loop where you just keep running other tests. In this mode, you should ignore other tests entirely until it works.