canary-test-authoring

name	canary-test-authoring
description	Use this skill when authoring or modifying Canary tests or job definitions, especially Python .pyt tests, parameters, dependencies, assets, artifacts, and resource requirements.

Authoring Canary tests

What this skill is for

Use this skill when you need to write, modify, or review Canary tests or job definitions.

Canary tests are generated by job generators. The most common reference format is a Python .pyt file handled by the Python job-definition generator, but Canary can also run jobs from other generators such as VVTest or CTest integrations.

First action: query test capabilities

Before authoring, query the targeted capability topic:

canary query -c tests

For more detail:

canary query -c tests.authoring
canary query -c tests.parameters
canary query -c dependencies
canary query -c resources

Recommended authoring workflow

1. Determine the job-definition format.

   • For .pyt, use Python directives and a guarded executable body.
   • For other formats, query or inspect the extension documentation.

2. Define metadata and generation behavior.

   • Use keywords for selection.
   • Use parameters for variants.
   • Use dependencies for workflow ordering.
   • Use resource directives or resource parameters for CPU/GPU/node needs.

3. Keep executable logic guarded.

   • Top-level directives are read during generation.
   • Runtime test logic should be under if __name__ == "__main__":.

4. Emit structured measurements when possible.

   • A test or post-test hook can record measurements.
   • Agents can later query them without parsing logs.

5. Run and inspect the generated jobs.

   • Use canary find or canary describe to inspect generation.
   • Use canary run to execute.

Minimal .pyt shape

import sys
import canary

canary.directives.keywords("smoke")


def test() -> int:
    return 0


if __name__ == "__main__":
    sys.exit(test())

For exact directive details, query:

canary query -c tests.directives

or use the generated directive reference in the Canary documentation.

Parameterized tests

Use parameters when you need multiple job variants.

Before changing parameter behavior, query:

canary query -c tests.parameters

Important rules:

• Parameter names appear in job names.
• cpus, gpus, and nodes have resource meaning.
• Any parameter matching a resource type in the resource pool may become resource-consuming.
• Fixed resource directives such as cpus(4) set resource needs without creating a named parameter variant.

Dependencies

Use dependencies when one job must wait for another.

Query:

canary query -c dependencies

Remember:

• Dependency selectors can match names, full names, paths, IDs, or glob patterns.
• Dependency conditions control whether downstream jobs run after upstream success, failure, or any completion.
• A downstream job can become BLOCKED if a dependency finishes with an unsatisfied outcome.

Assets and artifacts

Use assets for inputs that must be present in the job working directory.

Use artifacts for outputs that should be preserved or reported.

Query:

canary query -c tests.assets_and_artifacts

Common mistakes

• Putting runtime side effects at module top level in .pyt files.
• Expecting a dependency to match the source filename instead of the generated job name.
• Forgetting that parameterized jobs have expanded names.
• Using resource-like parameters without checking the resource pool.
• Parsing output logs instead of recording structured measurements.

After authoring

Run a narrow check first:

canary find -r PATH
canary describe PATH_OR_SPEC
canary run PATH

Then inspect:

canary status
canary log JOBID
canary query -j JOBID measurements

Use canary-run-debug for execution and debugging workflows.