// Development workflow for Security Verifiers. Use when asked to run tests, lint code, format files, set up the development environment, or perform CI checks on the codebase.
Run and analyze Security Verifiers evaluations. Use when asked to evaluate models on E1 (network-logs) or E2 (config-verification), generate metrics reports, compare model performance, or analyze eval results.
Generate SV-Bench metrics reports (summary.json + report.md) for E1/E2 runs, validate metrics contracts, and produce comparison-friendly artifacts from outputs/evals/.
Deploy Security Verifiers environments and packages. Use when asked to deploy to Prime Intellect Environments Hub, publish to PyPI, bump versions, build wheels, or manage releases.
Build and manage Security Verifiers datasets. Use when asked to build E1 or E2 datasets, create test fixtures, validate data, or manage dataset files for network-logs or config-verification environments.
Manage HuggingFace datasets for Security Verifiers. Use when asked to push datasets to HuggingFace, manage metadata, configure gated access, or set up user HF repositories for E1/E2 datasets.
| name | sv-dev |
| description | Development workflow for Security Verifiers. Use when asked to run tests, lint code, format files, set up the development environment, or perform CI checks on the codebase. |
| metadata | {"author":"security-verifiers","version":"1.0"} |
Testing, linting, formatting, and development workflow for the Security Verifiers codebase.
# Full setup (creates venv, installs all deps)
make setup
source .venv/bin/activate
# Run all checks
make check # lint + format + test
make test
make test-env E=network-logs
make test-env E=config-verification
make test-env E=code-vulnerability
make test-env E=phishing-detection
make test-env E=redteam-attack
make test-env E=redteam-defense
make e1 # test network-logs
make e2 # test config-verification
make e3 # test code-vulnerability
make e4 # test phishing-detection
make e5 # test redteam-attack
make e6 # test redteam-defense
make test-utils
make test-cov
uv run pytest environments/sv-env-network-logs/sv_env_network_logs_test.py::TestNetworkLogParser::test_extracts_label_and_confidence -q
make lint
make lint-fix
make format
make quick-fix
# Install and run hooks
make pre-commit
This runs:
Run the same checks as CI:
make ci
This runs:
ruff check . --exit-non-zero-on-fixpytest -q --tb=shortmake setup
Creates .venv, installs all environments and dev tools.
make venv # Create virtual environment
make install # Install all environments
make install-dev # Install dev tools (pytest, ruff, etc.)
For E2 config-verification, install pinned tool versions:
make install-linux # Installs kube-linter, opa, semgrep
make check-tools # Verify versions match ci/versions.txt
security-verifiers/
├── environments/ # Environment packages
│ ├── sv-env-network-logs/
│ ├── sv-env-config-verification/
│ └── ...
├── sv_shared/ # Shared utilities (security-verifiers-utils)
├── scripts/ # Build, eval, data scripts
├── outputs/ # Evaluation outputs
└── skills/ # Agent skills
make check # or make quick-check
make test-env E=network-logs # test specific env
make lint # check linting
make test-utils
make test # run all tests to check for regressions
make clean # Build artifacts and caches
make clean-outputs # Eval outputs (preserves logs)
make clean-logs # Log files only
make clean-all # Everything including venv
# Install: brew install entr (macOS) or apt install entr (Ubuntu)
make watch
Automatically runs tests when Python files change.
make info # Show environment status
make list-envs # List environment names
venv issues: make clean-all && make setup
Import errors: Ensure source .venv/bin/activate
Tool version mismatch: make check-tools then make install-linux
Pre-commit fails: make lint-fix && make format