| name | python-test-runner |
| description | Guidelines and instructions for building prerequisite example applications and running python-based integration and certification tests located in `src/python_testing/`. Use this skill to run important regression tests during developement or, when building new example apps, or when there is a need to execute python test scripts or "certification test scripts" or mentions of run_python_test.py and local.py. |
Python Test Runner
This skill provides expert guidelines for running Python-based integration and
certification tests located in src/python_testing/.
Prerequisite 1: Activating the Python Environment
Before running any Python tests, you must compile (if one doesn't exist in out/)
and activate the project's Python virtual environment.
[!IMPORTANT] Different runner methods may expect different virtual
environments. The modern local runner script (local.py) hardcodes out/venv
as its Python environment. Ensure you compile and activate this specific
folder:
./scripts/build_python.sh -i out/venv --enable_ipv4 true
source out/venv/bin/activate
Be sure to check the arguments in the build_python.sh script before setting up
the virtual environment. Run the shell script with the --help argument to see
all available options that may be relevant for the tests.
Prerequisite 2: Building the Example Application
Matter Python tests execute against a compiled device simulator application.
Before you can run any test locally, you must identify, map, and compile the
required application from source.
[!NOTE] For many modern Python tests, the use of the all-devices-app is
preferred, which allows simulating specific device type(s) on endpoints.
However, many existing tests continue to use chip-all-clusters-app, and
some domain-specific tests may require other specialized apps (e.g.,
chip-energy-management-app).
1. Identify the Required Application
Open the Python test file (e.g., src/python_testing/TC_FAN_3_3.py) and locate
the === BEGIN CI TEST ARGUMENTS === block near the top of the file. Under
the app parameter, you will see the application defined as an environment
variable, such as:
app: ${ALL_CLUSTERS_APP}app: ${ALL_DEVICES_APP}
These environment variables map to specific compiled applications. Locally, you
will need to know where these apps are built (usually in the out/ directory)
to reference them correctly.
2. Map Environment Variable to a Build Target
To compile the required application, match the environment variable from the
test header to a buildable target defined in scripts/build/build/targets.py.
Typical mappings:
${ALL_CLUSTERS_APP} -> Target: linux-x64-all-clusters-clang (builds
chip-all-clusters-app)${ALL_DEVICES_APP} -> Target: linux-x64-all-devices-clang (builds
all-devices-app)
To see the full list of compile targets available in the project:
scripts/run_in_build_env.sh "./scripts/build/build_examples.py targets"
3. Build the Target
Activate the environment and use the build_examples.py tool to compile the
application.
source scripts/activate.sh
./scripts/build/build_examples.py --target {build_target} build
Example:
source scripts/activate.sh
./scripts/build/build_examples.py --target linux-x64-all-devices-clang build
Once compiled, the application binary will be located in its corresponding
target folder under out/ (e.g.
out/linux-x64-all-devices-clang/all-devices-app). Keep track of this local
path, as you will pass it explicitly to the execution commands.
Determining Test Arguments
The header comments at the top of each python test file define the parameters
used for both the example app and the script.
[!TIP] When running locally and manually passing arguments, ignore or remove
environment-variable-based arguments (like ${TRACE_APP} or
${TRACE_TEST_JSON}) from your command line. These are intended for CI
pipelines.
Excluding Arguments During Local Testing
When running tests manually on a local workstation, you can (and sometimes
should) omit certain CI-mandated arguments depending on your specific testing
goal:
- Omitting
--factory-reset and --commissioning-method: If you are
running multiple test scripts sequentially against the same active device
session or commissioning window, you should omit these flags. Factory
resetting or re-commissioning is unnecessary if the device is already
configured and commissioned on the controller.
- Reusing KVS Storage: If you wish to verify state persistence across
application restarts (e.g. simulating a power cycle), make sure you do not
pass
--factory-reset or delete your --KVS database files.
Identifying Test Header Parameters
The header comments at the top of each python test file define the parameters
used for both the example app and the script. You can understand what each
parameter does using this description:
app: Indicates the application to be used in the test.
- Example:
app: ${ALL_DEVICES_APP}
factory-reset: Determines whether a factory reset should be performed
before the test (i.e., wiping persistent KVS files).
- Example:
factory-reset: true
quiet: Sets the verbosity level of the test run. When set to true,
the test runner generates less output.
app-args: Specifies the arguments to be passed to the application
during the test.
- Example:
--discriminator 1234 --KVS kvs1
--discriminator: Specifies the discriminator value used by the
application during commissioning to uniquely identify the device during
the discovery phase. This will change the pairing code for the device--KVS: Specifies the path to the Key-Value Store (KVS) file. The
KVS is a persistent database used by the application to store state
(such as commissioned fabrics, node IDs, etc.) so that they survive
application restarts. Specifying a unique KVS path prevents state
interference between concurrent test runs.
app-ready-pattern: Regular expression pattern to match against the
application's stdout to determine when the application has completed
initialization. The test runner blocks script execution until this pattern
is found.
- Example:
"Manual pairing code: \\[\\d+\\]"
app-stdin-pipe: Path to a named pipe that the test runner can use to
pipe standard input command strings into the application process.
script-args: Specifies the CLI arguments to be passed to the test
script itself.
- Example:
--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021
Python Script Command Line Arguments
When running the test script directly (using Python) or passing arguments inside
--script-args, you can customize the execution. Use --help on any script to
get a full list of arguments. Key parameters include:
--storage-path: Configures the file path where local state and
credentials are persisted (e.g., --storage-path admin_storage.json).
Providing this argument avoids state conflicts between concurrent runs.
Defaults to admin_storage.json in the current working directory.--commissioning-method: Specifies the method to use for commissioning
(e.g. on-network). Because chip-tool and the python controller do not
share credentials, the python script must commission the simulator target.- Pairing Codes:
--discriminator (e.g. 1234)--passcode (e.g. 20202021)--qr-code--manual-code
--tests: Filters which test case(s) within the script to execute.--PICS: Path to the PICS XML file or directory defining supported
features.- Custom Script Arguments (PIXITs): Used for passing custom config values
to the script. Must be supplied in
key:value format:
--bool-arg (e.g. --bool-arg pixit_name:False)--int-arg--float-arg--string-arg--json-arg--hex-arg
Running Tests: Method 1 (CI Meta Runner - Recommended)
The project provides a high-level local test harness script,
./scripts/tests/local.py. This is the preferred and most automated method
for running python tests locally because it pulls test metadata and CI arguments
directly from the headers of the Python test scripts.
Setup Requirements
This script hardcodes the virtual environment location to out/venv. You must
compile and activate this specific virtual environment path before running the
command:
./scripts/build_python.sh -i out/venv --enable_ipv4 true
source out/venv/bin/activate
Executing Tests
Use the python-tests subcommand and specify a filter using --test-filter to
target specific test suites:
./scripts/tests/local.py python-tests --test-filter {test_name}
Mapping Apps using --override-binary-path
Because the script reads the CI blocks directly, it parses variables like
app: ${ALL_CLUSTERS_APP}. By default, it attempts to look for built binaries
in default output folders. If your compiled binaries are located in a
non-standard build path, you must map the CI variable to your local compiled
file using --override-binary-path:
./scripts/tests/local.py python-tests \
--test-filter {test_name} \
--override-binary-path {CI_APP_VARIABLE} {path_to_local_built_binary}
Execution Example:
./scripts/tests/local.py python-tests \
--test-filter TC_RR_1_1 \
--override-binary-path ALL_CLUSTERS_APP out/linux-x64-all-clusters-no-ble/chip-all-clusters-app
Notable Arguments for local.py python-tests
--test-filter: Runs only the test scripts that match this glob pattern
(e.g., TC_RR_1_1 or TC_FAN_*).--override-binary-path: Maps a CI environment variable (e.g.,
ALL_CLUSTERS_APP, ALL_DEVICES_APP) to a specific path where the
application is built. Can be passed multiple times.--fail-log-dir: Specifies a directory to write logs when a test fails
(creates {test_name}.out.log and {test_name}.err.log inside the
directory). If not specified, failed test logs are printed directly to
stdout/stderr.--skip: Excludes specific test names or patterns from running.--include-nightly: Includes slow nightly tests that are skipped by
default.--keep-going: Instructs the runner to continue executing subsequent
tests even if a previous test in the queue fails.--help: View the help dialog for more information.
Running Tests: Method 2 (Combined Runner)
This uses the run_python_test.py script directly to manage starting the app,
running the test, and collecting logs. Unlike local.py, you must explicitly
provide all arguments since it does not read metadata blocks from the files.
Combined Runner Template
With the Python environment active:
./scripts/tests/run_python_test.py \
--factory-reset \
--app {path_to_compiled_app_binary} \
--app-args "{app_arguments}" \
--script {path_to_python_script} \
--script-args "{script_arguments}"
Combined Examples
Example A: Contact Sensor (All-Devices) Test
./scripts/tests/run_python_test.py \
--factory-reset \
--app ./out/linux-x64-all-devices-clang/all-devices-app \
--app-args "--device contact-sensor:1 --discriminator 1234 --KVS kvs1" \
--script src/python_testing/TC_IDM_2_3.py \
--script-args "--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --PICS src/app/tests/suites/certification/ci-pics-values"
Note that the --endpoint script argument is relevant for most apps and tests.
Some tests may exclude an endpoint argument and use the default one (which often
is EP 0), but it is best to explicitly specify which endpoint should be used.
Example B: All Clusters App Test
./scripts/tests/run_python_test.py \
--factory-reset \
--app ./out/linux-x64-all-clusters-clang/chip-all-clusters-app \
--app-args "--discriminator 1234 --KVS kvs1" \
--script src/python_testing/TC_GC_2_2.py \
--script-args "--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --endpoint 1"
Running Tests: Method 3 (Two Terminals)
This method is useful for interactive debugging, allowing you to monitor the
application's stdout logs side-by-side with the test script runner.
Step A: Start the Example Application
In one terminal, run your locally built application binary:
./out/linux-x64-all-devices-clang/all-devices-app \
--device contact-sensor:1 \
--discriminator 1234 \
--KVS kvs1
Note that the all devices app should always specify a device type and endpoint,
in the format device-type:endpoint-number
Step B: Run the Python Test Script
In a second terminal, run the python script after activating the Python virtual
environment:
source out/venv/bin/activate
python3 src/python_testing/TC_IDM_2_3.py \
--storage-path admin_storage.json \
--commissioning-method on-network \
--discriminator 1234 \
--passcode 20202021 \
--PICS src/app/tests/suites/certification/ci-pics-values
