메뉴
Run a local dry-run of an AI automation agent against real ADO data (no posts/commits made). Use to verify end-to-end connectivity and agent output before going live. Requires ADO PAT and LLM credentials in .env.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Verify a story's implementation on AEM as the QA Agent — navigate pages, check component rendering and dialogs, capture screenshots, and create Bug tickets for issues found. Requires AEM running locally. Use after deploying code.
Run accessibility testing on a URL using the axe MCP Server — analyze violations, get remediation guidance, apply fixes, and verify. Use when asked to check accessibility, run a11y audit, or fix WCAG issues.
Apply a small AEM change (a11y label, color, spacing, copy, css-class, icon, focus trap, or other small behavior tweak) by splitting work into authoring (JCR writes) and code (file edits → PR) paths. Reads the ADO story directly — a structured ```simple``` block is recommended but optional. Trigger on "simple change", "small tweak", "apply tweak".
Find all source files, AEM pages, and dialog fields for an AEM component. Use when a developer asks "where is component X?", "what files for hero?", or any component lookup question.
Generate AEM demo documentation — find or create docs page with configured component, capture dialog and website screenshots on QA, write authoring guide. Extends /aem-editorial-guide for automated pipeline use. Invoked by /dx-agent-all Phase 7 and /dx-req-dod.
Check health of AEM project infrastructure — verifies component definitions, OSGi configs, dispatcher rules, and content structure against expected state. Use to diagnose configuration drift or after making infrastructure changes.
| name | auto-test |
| description | Run a local dry-run of an AI automation agent against real ADO data (no posts/commits made). Use to verify end-to-end connectivity and agent output before going live. Requires ADO PAT and LLM credentials in .env. |
| when_to_use | Use to dry-run an AI automation agent against real ADO data before going live, when the user says 'test automation', 'dry run agent', 'verify end-to-end', or 'preview what the agent would do'. |
| argument-hint | [dor|dod|dod-fix|pr-review|pr-answer|bugfix|qa|devagent|docagent|estimation] [id] [[repoName]] [--dryRun] |
You run an AI automation agent locally against real ADO data. With --dryRun, results are logged but nothing is posted to ADO or committed. Without --dryRun, the agent runs fully (posts comments, applies fixes).
Check required env vars are set (in shell or .claude/settings.local.json):
ANTHROPIC_API_KEYADO_MCP_AUTH_TOKEN setFrom the user's command:
dor, dod, dod-fix, pr-review, pr-answer, bugfix, qa, devagent, docagent, or estimation--dryRun flag: default true for safetyIf no arguments: Read .ai/automation/infra.json to get automationProfile and the list of enabled agents (those without "disabled": true). Then prompt with only the enabled agents:
Which agent to test?
Only show agents that are enabled for this profile:
pr-only/pr-delegation): PR Review, PR Answer, DevAgent, BugFix, DoD FixFull agent list (filter to enabled):
- DoR — needs work item ID
- DoD — needs work item ID
- DoD Fix — needs work item ID
- PR Review — needs PR ID + repo name
- PR Answer — needs PR ID + repo name
- BugFix — needs Bug work item ID
- QA — needs work item ID
- DevAgent — needs work item ID (User Story)
- DOCAgent — needs work item ID (User Story)
- Estimation — needs work item ID
If the user requests an agent that is disabled for this profile, report: ⚠ <agent> is not enabled for the <profile> profile. To enable it, re-run /auto-init.
Target ID? (work item ID or PR ID)
Dry run? (default: yes — log results without posting to ADO)
- Yes — dry run (recommended for first test)
- No — live run (posts real comments/applies fixes)
node .ai/automation/scripts/pipeline-agent.js "<skill-prompt>"
Map agent names to skill prompts:
| Agent | Skill Prompt |
|---|---|
| dor | /dx-req-dod <id> |
| dod | /dx-req-dod <id> |
| dod-fix | /dx-req-dod <id> |
| pr-review | /dx-pr-review <pr-url> analyze only (dry run) or /dx-pr-review <pr-url> (live) |
| pr-answer | /dx-pr-answer <pr-url> |
| bugfix | /dx-bug-all <id> |
| devagent | /dx-agent-all <id> |
| docagent | /dx-doc-gen <id> |
| estimation | /dx-estimate <id> |
Stream output as it runs. This may take 1-3 minutes depending on the agent.
After completion, look for in the output:
[DOR] Posted DoR comment — DoR ran and posted[DOD] Posted DoD comment — DoD ran and posted[DOD-FIX] Applied fixes — DoD Fix ran and applied auto-fixes[PR-Review] Posted review — PR Review ran[PR-Answer] Posted reply — PR Answer ran[BUGFIX] Applied fix — BugFix ran[QA] Created Bug tickets — QA ran[DEVAGENT] Pipeline complete — DevAgent ran[DOCAGENT] Pipeline complete — DOCAgent ran--dryRun: true in logs — confirms dry run mode (nothing actually posted)[ALERT:CRITICAL] lines — investigate before going livestatus: skipped (dedupe/rate-limit) — expected on re-runs## Local Test Results
**Agent:** <agent>
**Target:** <id> (<work item title or PR title>)
**Mode:** Dry run / Live run
**Status:** ✓ Completed / ✗ Failed
<Key output lines>
<If dry run and succeeded:>
✓ Agent ran successfully in dry-run mode. Ready to go live.
Run `/auto-test <agent> <id> --live` to post real results.
<If failed:>
⚠️ Check the error above. Common issues:
- ADO PAT expired → rotate in AWS Lambda env vars (`/auto-lambda-env`)
- LLM API key invalid → check pipeline variables
- Network connectivity → check AWS Lambda VPC config
[ALERT:CRITICAL] errors/auto-test pr-review 12345 --dryRun — Runs the PR Review agent against PR #12345 in dry-run mode. Fetches the PR from ADO, runs the review logic locally, and prints the review output without posting any comments. Reports success — agent is ready for live use.
/auto-test dor 2416553 — Runs the DoR (Definition of Ready) agent against work item #2416553 in dry-run mode (default). Checks story completeness, acceptance criteria quality, and technical feasibility. Prints the assessment without updating the work item.
/auto-test pr-review 12345 --live — After confirming with user, runs the PR Review agent in live mode. Posts actual review comments to the PR in ADO. Use only after a successful dry-run.
"ADO PAT expired" error during test
Cause: The ADO Personal Access Token used by the Lambda has expired.
Fix: Rotate the PAT in ADO, then update it with /auto-lambda-env. For local dry-run testing, ensure the PAT is set in the pipeline variables or environment.
"LLM API key invalid" error
Cause: The Anthropic API key is missing or expired.
Fix: Check the pipeline variables for the LLM API key. Update with /auto-lambda-env if running against Lambda, or check local environment variables for dry-run mode.
Test passes in dry-run but fails in live mode Cause: Live mode makes additional ADO API calls (posting comments, updating work items) that may require different permissions than read-only dry-run. Fix: Ensure the ADO PAT has write permissions for work items and PR comments. Check the error message for the specific API call that failed.