Create/modify/debug platform check. Trigger: "platform check", "platform-checks", "upgrade check", "restart check", or writing Check class testing feature survival across restarts/upgrades. Also edits in misc/python/materialize/checks/all_checks/.
Create/modify/debug platform check. Trigger: "platform check", "platform-checks", "upgrade check", "restart check", or writing Check class testing feature survival across restarts/upgrades. Also edits in misc/python/materialize/checks/all_checks/.
Platform Checks
The Platform Checks framework is "write once, run everywhere" - a single test validates a feature across multiple scenarios (restarts, upgrades, etc.).
Key Concepts
Check: A Python class that tests one feature via testdrive fragments.
Scenario: The context (e.g., restart, upgrade) in which checks run.
Action: An individual step within a scenario (e.g., stop a container).
Where Checks Live
All checks are in misc/python/materialize/checks/all_checks/. The __init__.py auto-discovers every .py file in that directory, so adding a new file is all that's needed for registration.
Anatomy of a Check
A check is a class that extends Check and implements three methods. Each returns a Testdrive object wrapping a testdrive fragment string.
from textwrap import dedent
from materialize.checks.actions import Testdrive
from materialize.checks.checks import Check
classMyFeature(Check):
definitialize(self) -> Testdrive:
Testdrive(dedent())
() -> [Testdrive]:
[
Testdrive(dedent()),
Testdrive(dedent()),
]
() -> Testdrive:
Testdrive(dedent())
"""Run once at the start. Create tables, insert seed data, set up objects."""
"""MUST return a list of exactly 2 Testdrive fragments.
Each runs after a restart/upgrade phase. Insert more data, ALTER objects,
or create derived objects (e.g., materialized views)."""
return
"""
> INSERT INTO my_feature_table VALUES (4), (5);
> CREATE MATERIALIZED VIEW my_feature_view1 AS
SELECT COUNT(*) AS c FROM my_feature_table;
"""
"""
> INSERT INTO my_feature_table VALUES (6), (7);
> CREATE MATERIALIZED VIEW my_feature_view2 AS
SELECT SUM(f1) AS s FROM my_feature_table;
"""
def
validate
self
"""Run after all initialize + manipulate steps. Verify everything survived.
MAY BE CALLED MORE THAN ONCE - must be idempotent. Any objects created
here must be TEMPORARY or explicitly dropped."""
return
"""
> SELECT * FROM my_feature_view1;
7
> SELECT * FROM my_feature_view2;
28
"""
Both manipulate phases always run. validate() may run multiple times.
Decorators and Version Gating
Disable a check
from materialize.checks.checks import disabled
@disabled(ignore_reason="due to SQL-450")classMyBrokenCheck(Check):
...
Non-idempotent external interactions
If a check ingests non-UPSERT data into Kafka/Postgres (i.e., replaying manipulate would duplicate data), annotate it:
from materialize.checks.checks import externally_idempotent
@externally_idempotent(False)classMyKafkaCheck(Check):
...
Version-gated checks
For features that only exist from a certain version, override _can_run:
from materialize.checks.executors import Executor
from materialize.mz_version import MzVersion
classMyNewFeature(Check):
def_can_run(self, e: Executor) -> bool:
returnself.base_version >= MzVersion.parse_mz("v0.80.0-dev")
Version-conditional testdrive lines
Use [version>=VVMMPP] or [version<VVMMPP] prefixes for syntax that changed between versions (e.g., v0.92.0 → 9200):
>[version<9200] CREATE SOURCE s FROM LOAD GENERATOR COUNTER (SCALE FACTOR 0.01)
>[version>=9200] CREATE SOURCE s FROM LOAD GENERATOR COUNTER
Tips for Good Checks
Naming: Prefix ALL object names with a unique string (e.g., my_feature_) to avoid collisions with other checks running in the same Mz instance.
manipulate() returns exactly 2 elements: The framework asserts this.
validate() is idempotent: It may run multiple times. Use TEMPORARY objects or DROP at end.
Create objects in manipulate, not just initialize: This tests that DDL works after restarts/upgrades.
Create derived objects on both old and new base objects: e.g., a materialized view on a table from initialize AND on a table from manipulate.
Exercise full SQL syntax breadth: Test every WITH option, every variant of the feature.
Ingest data in every phase: Ensure data flows during restarts/upgrades.
Use dedent() on all testdrive strings: Required for proper testdrive parsing.
Helper Methods on Check
self.base_version: The MzVersion the test started from (useful for upgrade scenarios).
self.current_version: The MzVersion currently running.
self._kafka_broker(): Returns the broker connection string for testdrive.
self._default_cluster(): Returns "quickstart".
self.rng: A seeded Random instance for deterministic randomness.
Running
# Run a specific check against a specific scenario
bin/mzcompose --find platform-checks run default --scenario=RestartEntireMz --check=MyFeature
# Run with no restarts (useful for debugging the check itself)
bin/mzcompose --find platform-checks run default --scenario=NoRestartNoUpgrade --check=MyFeature
# Run all checks (default)
bin/mzcompose --find platform-checks run default --scenario=RestartEntireMz
Debugging CI Failures
Find the scenario name near the top of the log: Running ... --scenario=RestartEnvironmentdClusterdStorage
Find the failing check just before the error: Running validate() from <materialize.checks.threshold.Threshold ...>
Reproduce locally: bin/mzcompose --find platform-checks run default --scenario=RestartEnvironmentdClusterdStorage --check=Threshold
Try --scenario=NoRestartNoUpgrade to isolate whether the failure is restart/upgrade-related.