Comprehensive Agentforce testing skill with test execution, coverage analysis, and agentic fix loops. Run agent tests via sf CLI, analyze topic/action coverage, generate test specs, and automatically fix failing agents with 100-point scoring.
Comprehensive Agentforce testing skill with test execution, coverage analysis, and agentic fix loops. Run agent tests via sf CLI, analyze topic/action coverage, generate test specs, and automatically fix failing agents with 100-point scoring.
license
MIT
compatibility
Requires API v65.0+ (Winter '26) and Agentforce enabled org
metadata
{"version":"1.0.0","author":"Jag Valaiyapathy","scoring":"100 points across 5 categories"}
sf-ai-agentforce-testing: Agentforce Test Execution & Coverage Analysis
Expert testing engineer specializing in Agentforce agent testing, topic/action coverage analysis, and agentic fix loops. Execute agent tests, analyze failures, and automatically fix issues via sf-ai-agentforce.
Core Responsibilities
Test Execution: Run agent tests via sf agent test run with coverage analysis
Test Spec Generation: Create YAML test specifications for agents
Agent testing requires the Agent Testing Center feature, which is NOT enabled by default in all orgs.
Check if Agent Testing Center is Enabled
# This will fail if Agent Testing Center is not enabled
sf agent test list --target-org [alias]
# Expected errors if NOT enabled:# "Not available for deploy for this organization"# "INVALID_TYPE: Cannot use: AiEvaluationDefinition in this organization"
sf data query --use-tooling-api --query "SELECT Id FROM BotDefinition WHERE DeveloperName='X'"
Can't test non-existent agent
Agent published
sf agent validate authoring-bundle --api-name X
Must be published to test
Agent activated
Check activation status
Required for preview mode
Dependencies deployed
Flows and Apex in org
Actions will fail without them
Connected App (live)
OAuth configured
Required for --use-live-actions
Workflow (6-Phase Pattern)
Phase 1: Prerequisites
Use AskUserQuestion to gather:
Agent name/API name
Target org alias
Test mode (simulated vs live)
Coverage threshold (default: 80%)
Enable agentic fix loop?
Then:
Verify agent is published and activated
Check for existing test specs: Glob: **/*.yaml, Glob: **/tests/*.yaml
Create TodoWrite tasks
Phase 2: Test Spec Creation
Option A: Interactive Generation (no automation available)
# Interactive test spec generation
sf agent generate test-spec --output-file ./tests/agent-spec.yaml
# ⚠️ NOTE: There is NO --api-name flag! The command is interactive-only.
⭐⭐⭐⭐⭐ 90-100 pts → Production Ready
⭐⭐⭐⭐ 80-89 pts → Good, minor improvements
⭐⭐⭐ 70-79 pts → Acceptable, needs work
⭐⭐ 60-69 pts → Below standard
⭐ <60 pts → BLOCKED - Major issues
⛔ TESTING GUARDRAILS (MANDATORY)
BEFORE running tests, verify:
Check
Command
Why
Agent published
sf agent list --target-org [alias]
Can't test unpublished agent
Agent activated
Check status
Preview requires activation
Flows deployed
sf org list metadata --metadata-type Flow
Actions need Flows
Connected App (live)
Check OAuth
Live mode requires auth
NEVER do these:
Anti-Pattern
Problem
Correct Pattern
Test unpublished agent
Tests fail silently
Publish first: sf agent publish authoring-bundle
Skip simulated testing
Live mode hides logic bugs
Always test simulated first
Ignore guardrail tests
Security gaps in production
Always test harmful/off-topic inputs
Single phrasing per topic
Misses routing failures
Test 3+ phrasings per topic
CLI Command Reference
Test Lifecycle Commands
Command
Purpose
Example
sf agent generate test-spec
Create test YAML
sf agent generate test-spec --output-dir ./tests
sf agent test create
Deploy test to org
sf agent test create --spec ./tests/spec.yaml --target-org alias
sf agent test run
Execute tests
sf agent test run --api-name Test --wait 10 --target-org alias
sf agent test results
Get results
sf agent test results --job-id ID --result-format json
sf agent test resume
Resume async test
sf agent test resume --use-most-recent --target-org alias
sf agent test list
List test runs
sf agent test list --target-org alias
Preview Commands
Command
Purpose
Example
sf agent preview
Interactive testing
sf agent preview --api-name Agent --target-org alias
--use-live-actions
Use real Flows/Apex
sf agent preview --use-live-actions --client-app App
--output-dir
Save transcripts
sf agent preview --output-dir ./logs
--apex-debug
Capture debug logs
sf agent preview --apex-debug
Result Formats
Format
Use Case
Flag
human
Terminal display (default)
--result-format human
json
CI/CD parsing
--result-format json
junit
Test reporting
--result-format junit
tap
Test Anything Protocol
--result-format tap
Test Spec Quick Reference
Basic Template:
subjectType:AGENTsubjectName:<Agent_Name>testCases:# Topic routing-utterance:"What's on your menu?"expectation:topic:product_faqactionSequence: []
# Action invocation-utterance:"Search for Harry Potter books"expectation:topic:book_searchactionSequence:-search_catalog# Edge case-utterance:""expectation:graceful_handling:true
Impact: Tests created via YAML may have empty or mismatched expectations.
Topic Expectation Not Populated
Issue: When YAML includes topic: TopicName, the generated XML has empty expectedValue:
<!-- Generated (BROKEN) --><expectation><name>topic_assertion</name><!-- Missing expectedValue! --></expectation><!-- Should be --><expectation><name>topic_assertion</name><expectedValue>TopicName</expectedValue></expectation>
Agent Metadata Structure
Understanding agent components - Agent data is distributed across multiple metadata types:
Bot (Agent Definition)
└── BotVersion
└── genAiPlannerName → GenAiPlannerBundle
└── (references GenAiFunction for topics)
Retrieve all components:
sf project retrieve start --metadata "Bot:AgentName"
sf project retrieve start --metadata "GenAiPlannerBundle:AgentName_v1"
Note: BotDefinition is NOT queryable via Tooling API SOQL. Use metadata retrieval instead.
💡 Key Insights
Problem
Symptom
Solution
sf agent test create fails
"Required fields are missing: [MasterLabel]"
Use sf agent generate test-spec (interactive) or UI instead
Tests fail silently
No results returned
Agent not published - run sf agent publish authoring-bundle
Connected App not configured - use sf-connected-apps
Async tests stuck
Job never completes
Use sf agent test resume --use-most-recent
Empty responses
Agent doesn't respond
Check agent is activated
Agent Testing Center unavailable
"INVALID_TYPE" error
Use sf agent preview as fallback
Topic expectation empty
Test always passes topic check
Bug in CLI YAML→XML conversion; use interactive mode
⚠️ --use-most-recent broken
"Nonexistent flag" error on sf agent test results
Use --job-id explicitly - the flag is documented but NOT implemented
Topic name mismatch
Expected GeneralCRM, got MigrationDefaultTopic
Standard Salesforce copilots route to MigrationDefaultTopic - verify actual topic names from first test run
Test data missing
"No matching records" in outcome
Verify test utterances reference records that actually exist in org (e.g., "Edge Communications" not "Acme")
Action assertion fails unexpectedly
Expected [A], actual [A,B] but marked PASS
Action matching uses SUPERSET logic - actual can have MORE actions than expected and still pass
🔄 Two Fix Strategies
When agent tests fail, there are TWO valid approaches:
Agent Type
Fix Strategy
When to Use
Custom Agent (you control it)
Fix the agent via sf-ai-agentforce
Topic descriptions, action configurations need adjustment
Managed/Standard Agent (Salesforce copilot)
Fix test expectations in YAML
Test expectations don't match actual agent behavior
Decision Flow:
Test Failed → Can you modify the agent?
│
┌────────┴────────┐
↓ ↓
YES NO
↓ ↓
Fix Agent Fix Test Spec
(sf-ai-agentforce) (update YAML)
Example: Fixing Test Expectations
# BEFORE (wrong expectations)expectedTopic:GeneralCRMexpectedActions:-IdentifyRecordByName-GetRecordDetails# AFTER (matches actual behavior)expectedTopic:MigrationDefaultTopicexpectedActions:-IdentifyRecordByName-QueryRecords
🔄 Automated Test-Fix Loop
NEW in v1.1.0 | Claude Code can now orchestrate fully automated test-fix cycles
Overview
The test-fix loop enables Claude Code to:
Run tests → sf agent test run with JSON output
Analyze failures → Parse results and categorize issues
Fix agent → Invoke sf-ai-agentforce skill to apply fixes
Retest → Loop until all tests pass or max retries (3) reached
Escalate → Skip unfixable tests and continue with others
Quick Start
# Run the test-fix loop
./hooks/scripts/test-fix-loop.sh Test_Agentforce_v1 AgentforceTesting 3
# Exit codes:# 0 = All tests passed# 1 = Fixes needed (Claude Code should invoke sf-ai-agentforce)# 2 = Max attempts reached, escalate to human# 3 = Error (org unreachable, test not found, etc.)
Claude Code Integration
When Claude Code runs the test-fix loop:
USER: Run automated test-fix loop for Coral_Cloud_Agent
CLAUDE CODE:
1. bash hooks/scripts/test-fix-loop.sh Test_Agentforce_v1 AgentforceTesting
2. If exit code 1 (FIX_NEEDED):
- Parse failure details from output
- Invoke: Skill(skill="sf-ai-agentforce", args="Fix topic X: add keyword Y")
- Re-run: CURRENT_ATTEMPT=2 bash hooks/scripts/test-fix-loop.sh ...
3. Repeat until exit code 0 (success) or 2 (max retries)
Ralph Wiggum Integration (Hands-Off)
For fully automated loops without user intervention:
/ralph-wiggum:ralph-loop
> Run agentic test-fix loop for Test_Agentforce_v1 in AgentforceTesting until all tests pass
Issue: Different test creation methods use different expectation names:
CLI Generates
Manually Created Tests Use
topic_assertion
topic_sequence_match
actions_assertion
action_sequence_match
output_validation
bot_response_rating
Impact: May cause confusion when comparing test results from different sources.
Quick Start Example
# 1. Check if Agent Testing Center is enabled
sf agent test list --target-org dev
# 2. Generate test spec (automated)
python3 hooks/scripts/generate-test-spec.py \
--agent-file ./agents/MyAgent.agent \
--output ./tests/myagent-tests.yaml
# 3. Create test in org
sf agent test create \
--spec ./tests/myagent-tests.yaml \
--api-name MyAgentTest \
--target-org dev
# 4. Run tests
sf agent test run \
--api-name MyAgentTest \
--wait 10 \
--result-format json \
--target-org dev
# 5. View results
sf agent test results \
--use-most-recent \
--verbose \
--result-format json \
--target-org dev
