testing

name	testing
description	Run or troubleshoot IntelliJ `tests.cmd` tests and discovery.

Testing Guide for IntelliJ IDEA

Quick Start

./tests.cmd --module <module> --test <pattern>

• --module — the JPS module containing the test classes (always use the test's own module). To find the module name, look at the .iml file in the test's directory — the module name is the .iml filename without the extension.
• --test — FQN, wildcard pattern, or FQN#methodName

# Single test class (FQN)
./tests.cmd --module intellij.cidr.compiler.custom.tests \
            --test com.intellij.cidr.compiler.custom.CidrCustomCompilerReadTest

# Wildcard pattern
./tests.cmd --module intellij.goland.tests \
            --test com.goide.comments.*Test

# Specific test method (wildcards cannot be used with #)
./tests.cmd --module intellij.cidr.compiler.custom.tests \
            --test com.intellij.cidr.compiler.custom.CidrCustomCompilerReadTest#testSingleDefine

# Multiple (semicolon-separated)
./tests.cmd --module intellij.platform.build.tests \
            --test org.jetbrains.intellij.build.TestSelectorsTest#class selector;org.jetbrains.intellij.build.FileSetTest

Simple class names like MyTest do NOT work — always use FQN or wildcard (*MyTest).

Why simple names fail

Patterns are transformed (* → .*, . → \.) and matched against the fully qualified class name using Pattern.matches() (full-string match):

Pattern	Regex	Matches org.example.MyTest?
MyTest	MyTest	NO — doesn't cover the package prefix
*MyTest	.*MyTest	YES
org.example.MyTest	org\.example\.MyTest	YES
org.example.*	org\.example\..*	YES

Community tests

Use community/tests.cmd for tests that belong to community-only modules:

./community/tests.cmd --module <module> --test <pattern>

API checks via Bazel

In an Ultimate checkout, run ApiCheckTest directly through its Bazel test target. For a focused check, pass one or more comma-separated module names:

bazel test //tests/ideaProjectStructure:projectStructureTests_test \
  --test_filter=com.intellij.ideaProjectStructure.api.ApiCheckTest \
  --test_arg=--jvm_flag=-Dapi.dump.test.modules.to.check=<module>[,<module>...] \
  --test_output=summary \
  --test_summary=detailed

Omit --test_arg to check all modules. Do not use bazel run for this target: it streams noisy test output and does not provide Bazel's test summary.

On failure, keep agent context small by extracting the failing dynamic-test names and messages from Bazel's JUnit report instead of printing the full test.log:

xmllint --xpath '//testcase[failure or error]/@name | //testcase[failure or error]/*[self::failure or self::error]/@message' \
  out/bazel-testlogs/tests/ideaProjectStructure/projectStructureTests_test/test.xml

Product layout and packaging changes

When changing ProductProperties, productImplementationModules, product content descriptors, plugin/module-set packaging, or generated product layout XML, also run:

./tests.cmd --module intellij.idea.ultimate.build.tests --test com.intellij.idea.ultimate.build.smokeTests.AllProductsPackagingTest

Windows PowerShell note

When running tests.cmd from PowerShell, pass JVM -D... arguments via stop-parsing mode to avoid argument mangling:

./tests.cmd --% -Dintellij.build.test.patterns=com.example.MyTest

Without --%, PowerShell can alter -D... arguments before they reach tests.cmd, which may lead to errors like Could not find or load main class ....

Separate Bazel Modules

Some parts of the repository are standalone Bazel modules and must not use tests.cmd or community/tests.cmd.

community/platform/build-scripts/bazel

• This directory is a separate Bazel module.
• Tests in this module must be run via Bazel from that directory.
• Use the command documented in community/platform/build-scripts/bazel/README.md:

cd community/platform/build-scripts/bazel
../../../../bazel.cmd test //:bazel-generator-integration-tests --test_output=all

• Do not use ./tests.cmd for org.jetbrains.intellij.build.bazel.BazelGeneratorIntegrationTests or other tests in that module.
• If a request touches files under community/platform/build-scripts/bazel, prefer that module-local ../../../../bazel.cmd test flow for verification.

How tests.cmd Works

1. tests.cmd is a cross-platform script (works on Windows/Linux/macOS)
2. It takes --module and --test, maps them to JVM properties, and calls bazel run //build:local_idea_ultimate_run_tests_build_target
3. The test runner uses JUnit to execute the specified test classes

Troubleshooting test discovery:

• Bazel incremental compilation works correctly — remote caches do NOT cause staleness, don't waste time on bazel clean
• Test discovery issues are typically caused by:
  1. Wrong test pattern (use FQN or wildcard, not simple class name)
  2. Wrong module
  3. Test class not in the correct test module's classpath

For deeper troubleshooting, see TESTING-internals.md.

tests.cmd Parameters

Usage: tests.cmd --module <module> --test <pattern> [options]

Required:
  --module <module>    Name of the JPS module which contains the test classes
  --test <pattern>     Full test class name (FQN) or wild card pattern (e.g. com.intellij.*Test) or exact FQN#methodName

Options:
  --debug              Debug build scripts JVM process
  --help               Show this help message

Additional options are passed as JVM flags to org.jetbrains.intellij.build.TestingOptions
  Example: -Dintellij.build.test.debug.enabled=true -Dintellij.build.test.debug.suspend=true -Dintellij.build.test.debug.port=5005

Additional JVM options

Extra -D... arguments are passed through as JVM flags to org.jetbrains.intellij.build.TestingOptions:

-Dintellij.build.test.attempt.count=<n>

• Retry failed tests N times
• Default: 1 (no retries)
• Use 3 for flaky tests

-Dintellij.build.test.jvm.memory.options=<options>

• Custom JVM memory options for the test process
• Example: -Xmx8g for 8GB heap

-Dpass.<property>=<value>

• Pass arbitrary system properties to the test JVM
• The pass. prefix is stripped, so -Dpass.my.flag=true becomes -Dmy.flag=true in the test JVM

Debugging: -Dintellij.build.test.debug.enabled=true -Dintellij.build.test.debug.port=5005 -Dintellij.build.test.debug.suspend=true — attach IDE debugger to port 5005.

Troubleshooting

Tests fail with OutOfMemoryError:

• Increase heap size: -Dintellij.build.test.jvm.memory.options=-Xmx8g
• Check for memory leaks in test code

Tests not found:

• Verify --test uses FQN or wildcard, not simple class name: --test com.example.MyTest
• Check that --module is the module that actually contains the test class (look at the .iml file location)
• Check that class name ends with Test (or use -Dpass.idea.include.unconventionally.named.tests=true)
• Before troubleshooting further, check whether the test lives in a separate Bazel module such as community/platform/build-scripts/bazel; those tests must be run with module-local ../../../../bazel.cmd test, not tests.cmd

Tests pass locally but fail in CI:

• Check test isolation - tests may depend on execution order
• Verify environment variables and system properties
• Use -Dintellij.build.test.attempt.count=3 for flaky tests

Bazel build fails before tests run:

• Check module dependencies in .iml file
• Consult module-dependencies.md

For deeper troubleshooting, see TESTING-internals.md.

Test Execution Internals

For detailed information about how tests.cmd works internally, including:

• Execution flow diagrams
• Key classes reference
• TestingOptions properties
• Bazel target configuration
• Test discovery flow

See TESTING-internals.md

Writing Tests

For guidelines on writing tests (as opposed to running them), see the guidelines:

• Writing Tests - How to write tests, always consult before writing new tests

Additional Resources

• Running and Testing Documentation
• Community README
• Main README
• Driver UI testing