kalitool

name	kalitool
description	Run a user-specified DrowAI tool against a real Kali task container using the tool's real schema and safe placeholder targets, then write a markdown validation report. Use when the user asks to test a tool in real Kali, validate tool parameters from schema, verify runtime behavior through task/container execution, or batch-test the Kali tool matrix.

Real Kali Tool Schema Test

Purpose

Validate one user-provided tool end-to-end using:

• real tool schema from the registry,
• real task/container lifecycle,
• real execution path in Kali with no mock fallback.

Quick Start

Run the utility script from the repo root:

python .codex/skills/kalitool/scripts/run_real_kali_tool_schema_test.py --tool-id information_gathering.network_discovery.masscan --jwt-token "<JWT>"

Or use login-based auth:

python .codex/skills/kalitool/scripts/run_real_kali_tool_schema_test.py --tool-id information_gathering.network_discovery.nmap --username "<user>" --password "<pass>"

Use full schema coverage when validating optional parameters:

python .codex/skills/kalitool/scripts/run_real_kali_tool_schema_test.py --tool-id information_gathering.network_discovery.nmap --params full --jwt-token "<JWT>"

Non-Negotiable Rules

• Use strict real-Kali mode only. If Docker/Kali runtime is unavailable, fail and report.
• Never run against real external targets.
• Use safe placeholders:
  • IP-like fields -> 127.0.0.1
  • host/domain-like fields -> example.com, localhost, or acme.local
• Require authentication for task APIs via JWT bearer token.
• Output a markdown report file.
• Never log raw JWTs, passwords, cookies, or bearer tokens.

Required Inputs

• tool_id (required): exact tool registry id to test.
• Authentication (one required):
  • jwt_token, or
  • username + password (login first to obtain token).

Optional:

• params: minimal or full (default: minimal).
• report_path (default: artifacts/tool-schema-test-<tool_id>.md).
• keep_on_failure (default: false).
• api_base_url (default: http://localhost:8000).

Workflow

Use this checklist and keep it updated while running:

Progress
- [ ] 1) Validate inputs and auth strategy
- [ ] 2) Resolve JWT token
- [ ] 3) Create temporary task
- [ ] 4) Wait for container/runtime readiness
- [ ] 5) Load tool schema from registry
- [ ] 6) Build safe parameters
- [ ] 7) Execute tool via real Kali execution path
- [ ] 8) Collect and evaluate result
- [ ] 9) Write markdown report
- [ ] 10) Cleanup task/container

1) Validate inputs

• Fail immediately if tool_id is missing.
• Fail immediately if neither jwt_token nor username/password are present.

2) Resolve JWT token

• If jwt_token is provided, use it.
• Else call POST /api/auth/login and extract access_token.
• Use header: Authorization: Bearer <token>.
• Never log the raw token.

3) Create temporary task

• Call POST /api/tasks/ with minimal payload.
• Prefer a deterministic name prefix, e.g. skill-tooltest-<tool_id>-<timestamp>.
• Track returned task_id.

4) Wait for readiness

• Poll task/container status until task is active and container exists/runs.
• If startup fails or times out, stop and produce failure report.

5) Load real schema

• Fetch schema from tool registry metadata:
  • get_tool_metadata(tool_id)["args_schema"]
• Extract required fields and field types.

6) Build safe parameters

• Start with minimal valid schema-compliant payload.
• For --params full, include applicable optional fields with safe/default values.
• Override target-like fields with safe placeholders.
• Never allow user-provided real targets in this workflow.

Suggested mapping heuristics:

• key contains ip, address, src_ip, dst_ip -> 127.0.0.1
• key contains target, host, hostname, domain, url:
  • if field appears IP-typed -> 127.0.0.1
  • else -> example.com (or http://localhost if URL with scheme is required)

7) Execute in real Kali

• Use the real task/container execution path, the same runtime route used by the system.
• Do not substitute mock execution.
• Capture:
  • success
  • exit_code
  • stdout/stderr
  • metadata
  • validation errors, if present

8) Evaluate

PASS when schema validation succeeds and command execution reaches a valid tool result.

FAIL when any of:

• auth failure
• task/container startup failure
• schema validation error
• runtime execution error
• timeout

9) Write markdown report

Use the report template generated by scripts/run_real_kali_tool_schema_test.py.

10) Cleanup

• Default behavior: stop/remove runtime and delete temporary task.
• If keep_on_failure=true, preserve the task for debugging and mention this in the report.

Batch Testing

• Generate or refresh the batch state file:

python .codex/skills/kalitool/scripts/generate_kalitool_tool_state.py

• Run unchecked tools with both minimal and full parameters:

python .codex/skills/kalitool/scripts/run_batch_kalitool_tests.py --jwt-token "<JWT>"

Failure Policy

• Do not silently downgrade to mock or host-only execution.
• Do not continue after auth failure.
• Do not continue when safe-target enforcement cannot be guaranteed.

References

Open only what you need:

• Examples: references/examples.md
• Runtime scripts: scripts/run_real_kali_tool_schema_test.py, scripts/run_batch_kalitool_tests.py, scripts/generate_kalitool_tool_state.py