devkit-test

namespace	aiwg
name	devkit-test
platforms	["all"]
description	Auto-fix discoverable issues

Test AIWG Package

Run comprehensive tests on an AIWG package to verify structure, functionality, and deployment readiness.

Process

1. Locate and Identify Package

Resolve the package path from $ARGUMENTS:

• If relative path, resolve from current directory
• If package name, look in addons/ and frameworks/

Identify package type from manifest.json:

• "type": "addon" → Addon tests
• "type": "framework" → Framework tests
• "type": "extension" → Extension tests

2. Run Structure Tests

Manifest Validation:

• manifest.json exists and is valid JSON
• Required fields present: id, type, name, version, description
• Entry paths exist and match actual directories
• Component arrays match actual files

Directory Structure:

• Expected directories exist (agents/, commands/, etc.)
• No orphaned files outside expected structure
• README.md exists and is non-empty

Component Files:

• All agents listed in manifest exist as .md files
• All commands listed in manifest exist as .md files
• All skills listed in manifest have SKILL.md
• All templates listed in manifest exist

3. Run Content Tests

Agent Validation: For each agent file:

• Valid YAML frontmatter with name, description, model, tools
• Model is valid (sonnet, opus, haiku)
• Tools list contains valid tool names
• Content sections present (Domain Expertise, Responsibilities, etc.)

Command Validation: For each command file:

• Valid YAML frontmatter with name, description
• Args defined with name and description
• Process section present
• No broken internal references

Skill Validation: For each skill:

• SKILL.md has valid frontmatter
• Trigger phrases defined
• Execution process documented
• References directory exists (if referenced)

4. Run Type-Specific Tests

Addon Tests:

• No requires field (addons are standalone)
• Can be deployed independently

Extension Tests:

• requires field present and valid
• Parent framework exists
• Located in correct extensions/ subdirectory

Framework Tests:

• Phases defined in manifest
• Flow documents exist for each phase
• actors-and-templates.md exists
• config/models.json exists
• metrics/tracking-catalog.md exists

5. Run Deployment Test

Simulate deployment to temporary directory:

# Create temp test directory
mkdir -p /tmp/aiwg-test-$$

# Attempt deployment
aiwg -deploy-agents --source <package-path> --target /tmp/aiwg-test-$$ --dry-run

# Check expected output structure

6. Auto-Fix (if --fix)

When --fix is specified, attempt to repair:

Fixable Issues:

• Missing directories → Create them
• Manifest component arrays out of sync → Update from files
• Missing README.md → Generate from manifest
• Invalid JSON formatting → Reformat

Non-Fixable Issues (report only):

• Missing required fields in manifest
• Invalid agent/command content
• Missing parent framework (extensions)

7. Generate Report

Summary Format:

Package Test: <package-id>
Type: <addon|extension|framework>
─────────────────────────────────

Structure Tests: <pass/fail count>
Content Tests: <pass/fail count>
Type Tests: <pass/fail count>
Deployment Test: <pass/fail>

Overall: <PASS|FAIL>

Detailed Format (--verbose):

Package Test: <package-id>
─────────────────────────────────

[Structure Tests]
  ✓ manifest.json exists
  ✓ manifest.json is valid JSON
  ✓ Required fields present
  ✓ agents/ directory exists
  ✗ commands/ directory missing
  ...

[Content Tests]
  ✓ agent-one.md: valid frontmatter
  ✓ agent-one.md: valid model (sonnet)
  ✗ agent-two.md: missing tools field
  ...

[Type Tests]
  ✓ No requires field (addon)
  ...

[Deployment Test]
  ✓ Dry-run deployment successful

[Issues Found]
  ERROR: commands/ directory missing
  WARNING: agent-two.md missing tools field

[Auto-Fixed] (if --fix)
  ✓ Created commands/ directory

Exit Codes

• 0: All tests passed
• 1: Tests failed (fixable issues remain)
• 2: Tests failed (non-fixable issues)

Examples

# Test addon
/devkit-test aiwg-utils

# Test with verbose output
/devkit-test sdlc-complete --verbose

# Test and auto-fix
/devkit-test my-addon --fix --verbose

# Test extension
/devkit-test sdlc-complete/extensions/hipaa

References

• @$AIWG_ROOT/agentic/code/addons/aiwg-dev/README.md — aiwg-dev addon overview
• @$AIWG_ROOT/docs/extensions/overview.md — Extension system architecture and package types
• @$AIWG_ROOT/docs/cli-reference.md — CLI reference including validate-metadata command
• @$AIWG_ROOT/agentic/code/addons/aiwg-utils/rules/research-before-decision.md — Verification-before-action pattern
• @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/README.md — SDLC framework as example test target