troubleshoot

name	troubleshoot
description	Diagnose and fix common ABCA issues: deployment failures, preflight errors, authentication problems, agent failures, and build issues. Use when the user says "troubleshoot", "debug", "not working", "error", "failed", "help me fix", "preflight_failed", "task failed", "deploy failed", "auth error", "401", "422", "503", or describes something not working as expected.

ABCA Troubleshooting

You are diagnosing an issue with the ABCA platform. Follow a systematic approach: gather symptoms, check the most common causes, and apply targeted fixes.

Running the CLI: commands below call node cli/lib/bin/bgagent.js …. In a non-interactive or mise-managed shell node may not be on PATH — prefix with mise exec --. Ironically, node: command not found is itself a common symptom (the shell hasn't activated mise); that's a missing prefix, not a broken install.

Step 1: Identify the Problem Category

Determine which area the issue falls into:

1. Build/Compilation — TypeScript errors, test failures, lint issues
2. Deployment — CDK deploy/synth failures, CloudFormation errors
3. Authentication — Cognito errors, token issues, 401 responses
4. Task Submission — 422 errors, validation failures, guardrail blocks
5. Task Execution — Preflight failures, agent failures, timeouts
6. Local Agent Testing — Docker issues, run.sh problems

Build/Compilation Issues

export MISE_EXPERIMENTAL=1
mise //cdk:compile 2>&1 | tail -50  # TypeScript errors
mise //cdk:test 2>&1 | tail -50     # Test failures

Common causes:

• Missing mise run install after pulling changes
• yarn: command not found — Run corepack enable && corepack prepare yarn@1.22.22 --activate
• Type mismatches after editing cdk/src/handlers/shared/types.ts without updating cli/src/types.ts

Deployment Issues

# Check CloudFormation events for the failed stack
aws cloudformation describe-stack-events --stack-name backgroundagent-dev \
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED` || ResourceStatus==`UPDATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' \
  --output table

Common causes:

• Docker not running — Required for CDK asset bundling
• Missing CDK bootstrap — Run mise //cdk:bootstrap
• IAM permission issues — Check aws sts get-caller-identity
• Region mismatch — Ensure consistent region across all commands

Authentication Issues

# Verify credentials
aws sts get-caller-identity

# Check Cognito user exists
aws cognito-idp admin-get-user \
  --user-pool-id $USER_POOL_ID \
  --username [e-mail]

Common causes:

• "App client does not exist" — Region mismatch between CLI config and stack deployment
• Token expired — Re-authenticate with bgagent login
• 401 on API calls — Token not included or malformed in Authorization header
• User not created — Self-signup is disabled; admin must create users

Task Submission Issues (422 / 400)

"Repository not onboarded" / REPO_NOT_ONBOARDED (422):

• The repo isn't registered. Fastest fix: bgagent repo onboard <owner/repo> (operator path — writes the RepoTable record at runtime, no redeploy). A CDK Blueprint is only needed for declarative config. Use the onboard-repo skill for details.
• Also confirm the owner/repo matches exactly what you pass to bgagent submit --repo.

"GUARDRAIL_BLOCKED" (400):

• Task description triggered Bedrock Guardrails content screening
• Review and rephrase the task description to remove potentially flagged content

Validation errors:

• Check required fields: repo is required, plus at least one of issue_number, task_description, pr_number
• max_turns range: 1-500
• max_budget_usd range: $0.01-$100

Task Execution Issues

# Check task events for details
node cli/lib/bin/bgagent.js events <TASK_ID> --output json

preflight_failed:

• GitHub PAT lacks permissions for the repo
• Repository doesn't exist or is private without proper token scope
• Check event reason and detail fields for specifics
• Verify PAT: fine-grained token must include the target repository with Contents (read/write), Pull Requests (read/write), Issues (read)

task_failed / task completes with 0 tokens and no PR:

• Agent encountered an error during execution
• Check CloudWatch logs for the session:

  aws logs filter-log-events \
    --log-group-name "/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/jean_cloude" \
    --filter-pattern "<TASK_ID>" \
    --region us-west-2 --query 'events[*].message' --output text
  

• Common: repo build/test commands not documented in CLAUDE.md

403 "not authorized to perform bedrock:InvokeModelWithResponseStream":

• The repo's model_id is a model the runtime IAM role wasn't granted. The runtime only has grantInvoke for the models in the stack's configured set (Sonnet 4.6, Opus 4, Haiku 4.5 by default).
• Quick fix: point the repo at an already-granted model — bgagent repo onboard <owner/repo> --model us.anthropic.claude-sonnet-4-6 (no redeploy).
• To add a new model to the runtime: grant it in the stack and redeploy. The model set is the shared list in cdk/src/constructs/bedrock-models.ts — add the model via the bedrockModels CDK context (cdk.json) so both the AgentCore and ECS backends grant it (#433). Adding a model also requires account-level Bedrock access for it (separate from IAM — see the next row).

Model not enabled / "not available on your Bedrock deployment" (often immediate failure, few turns, zero or near-zero tokens):

• IAM is necessary but not sufficient. The AgentCore role may already have bedrock:InvokeModel*, but the account must also satisfy Amazon Bedrock model access: Marketplace subscription flow on first serverless use (with aws-marketplace:Subscribe / ViewSubscriptions where needed), Anthropic first-time use details (PutUseCaseForModelAccess or the console model catalog), and a valid payment method for Marketplace-backed models.
• Use an inference profile ID in the Blueprint / DynamoDB model_id when Bedrock requires it for on-demand invocation (for example us.anthropic.claude-sonnet-4-6 for US Sonnet 4.6). See Use an inference profile in model invocation. Raw anthropic.* IDs often hit "on-demand not supported" or wrong routing — see the 400 row below.
• Cross-Region profiles route across Regions in a geography; ensure IAM and any SCPs allow Bedrock in all destination Regions for that profile. See Supported Regions and models for inference profiles.
• Task status: When the Claude CLI reports a terminal error via ResultMessage.is_error, the agent marks the task FAILED (not COMPLETED) and persists error_message in DynamoDB.

400 "Invocation with on-demand throughput isn't supported":

• The Blueprint modelId uses a raw foundation model ID (e.g. anthropic.claude-opus-4-20250514-v1:0)
• Fix: change to the inference profile ID (e.g. us.anthropic.claude-opus-4-20250514-v1:0), update DynamoDB via redeploy

503 "Too many connections" / task completes with 0 tokens after long duration:

• Bedrock is throttling model invocations. The agent retries for minutes then gives up.
• Symptoms: task runs for 10-15 minutes, may end with COMPLETED if the SDK does not flag ResultMessage.is_error (unlike hard Bedrock entitlement errors, which surface as FAILED once the CLI sets is_error on the result)
• Diagnosis:
  1. Check application logs for "text": "API Error: 503 Too many connections"
  2. Check what model_id is actually being passed — the DynamoDB record may have a stale model override:

     aws dynamodb get-item \
       --table-name <RepoTableName> \
       --key '{"repo": {"S": "owner/repo"}}' \
       --query 'Item.model_id' --output text
     

• Causes:
  • Stale model_id in DynamoDB (most common) — the Blueprint onUpdate only sets fields present in props; removing a modelId prop does NOT remove the field from DynamoDB. The task keeps using the old model.
  • Bedrock service-level throttling for the specific model (especially Opus 4 which has limited availability)
  • Account quota limits reached
• Fix:
  1. Check and fix the DynamoDB record first — remove stale model_id if present:

     aws dynamodb update-item \
       --table-name <RepoTableName> \
       --key '{"repo": {"S": "owner/repo"}}' \
       --update-expression "REMOVE model_id"
     

  2. If model_id is correct, wait and retry — throttling is often transient
  3. Switch to a model with higher availability (Sonnet 4.6 > Opus 4 > Haiku)
  4. Request a Bedrock quota increase for InvokeModel RPM on your model

task_timed_out:

• 9-hour maximum exceeded
• Consider reducing scope or increasing max_turns for complex tasks
• Check if the agent is stuck in a loop (review logs)

Concurrency limit:

• Default: 3 concurrent tasks per user
• Wait for running tasks to complete or cancel them

Local Agent Testing Issues

# Verify Docker is running
docker info

# Test locally with dry run
DRY_RUN=1 ./agent/run.sh "owner/repo" "Test task"

Common causes:

• Missing environment variables: GITHUB_TOKEN, AWS_REGION
• Docker not running or insufficient resources (needs 2 vCPU, 8 GB RAM)
• Missing AWS credentials for Bedrock access

Diagnostic Commands Quick Reference

# Stack status
aws cloudformation describe-stacks --stack-name backgroundagent-dev --query 'Stacks[0].StackStatus'

# Stack outputs
aws cloudformation describe-stacks --stack-name backgroundagent-dev --query 'Stacks[0].Outputs' --output table

# Task status (use --verbose for HTTP-level debug output)
node cli/lib/bin/bgagent.js --verbose status <TASK_ID>
node cli/lib/bin/bgagent.js events <TASK_ID> --output json

# Watch task progress in real time
node cli/lib/bin/bgagent.js watch <TASK_ID>

# Download full execution trace (task must have been submitted with --trace)
node cli/lib/bin/bgagent.js trace download <TASK_ID>

# List running tasks
node cli/lib/bin/bgagent.js list --status RUNNING

# Build health
mise run build

Tip: Add --verbose to any bgagent command to see the full HTTP request/response cycle on stderr. This is the fastest way to diagnose auth, network, or API contract issues.