| name | ai-performance-testing |
| description | Use when working with an AI tool to define and measure factual accuracy (>95% correct citations), completeness (>90% complete answers), and consistency (>85% consistency score) of AI systems. |
| metadata | {"author":"dtmc-marketplace"} |
AI Performance Testing Skill
This skill provides a structured workflow for performance testing of AI systems, focusing on factual accuracy, completeness, and consistency. It leverages the Deepeval framework to measure the performance of LLM-based systems.
Workflow for AI Performance Testing
Follow these steps to conduct comprehensive AI performance testing:
Remind: you should create new files instead of modify template
Remind: you should use local existing venv instead of directly create one
Remind: you should only modify the tested model, do not change any config for the testing process/models
-
Generate Test Data:
- Objective: Create a dataset for testing the AI system.
- Action: Use the
template/data_generator.py script to generate test data. You may need to ask the user for a data source to integrate. The output data file should be appropriately named for the test run.
- Verification: A test data file is created and available for the performance test.
-
Set up the Target LLM-based System:
- Objective: Integrate the LLM-based system to be tested with the Deepeval framework.
- Action: Configure the
template/performance_test.py script to connect to the target LLM system. This may involve setting API keys, model names, and other parameters.
- Verification: The setup is complete and the system is ready for testing.
-
Execute Performance Tests:
- Objective: Run the performance tests to measure the system's factual accuracy, completeness, and consistency.
- Action: Execute the
template/performance_test.py script. The script will use the generated test data to query the LLM system and evaluate its responses against the defined metrics.
- Verification: The test script runs successfully and outputs the performance metrics.
-
Generate Performance Report:
- Objective: Create a structured report summarizing the test findings.
- Action: Process the test results to generate a markdown-formatted report that includes detailed outcomes for accuracy, completeness, and consistency.
- Verification: Ensure the report is comprehensive, accurate, and saved as a
.md file in an accessible location.
Converted and distributed by TomeVault — claim your Tome and manage your conversions.
