// Manage HuggingFace datasets for Security Verifiers. Use when asked to push datasets to HuggingFace, manage metadata, configure gated access, or set up user HF repositories for E1/E2 datasets.
Run and analyze Security Verifiers evaluations. Use when asked to evaluate models on E1 (network-logs) or E2 (config-verification), generate metrics reports, compare model performance, or analyze eval results.
Generate SV-Bench metrics reports (summary.json + report.md) for E1/E2 runs, validate metrics contracts, and produce comparison-friendly artifacts from outputs/evals/.
Deploy Security Verifiers environments and packages. Use when asked to deploy to Prime Intellect Environments Hub, publish to PyPI, bump versions, build wheels, or manage releases.
Build and manage Security Verifiers datasets. Use when asked to build E1 or E2 datasets, create test fixtures, validate data, or manage dataset files for network-logs or config-verification environments.
Development workflow for Security Verifiers. Use when asked to run tests, lint code, format files, set up the development environment, or perform CI checks on the codebase.
| name | sv-hf |
| description | Manage HuggingFace datasets for Security Verifiers. Use when asked to push datasets to HuggingFace, manage metadata, configure gated access, or set up user HF repositories for E1/E2 datasets. |
| metadata | {"author":"security-verifiers","version":"1.0"} |
Push, validate, and manage datasets on HuggingFace Hub for E1 (network-logs) and E2 (config-verification) environments.
| Repo Type | E1 Repo | E2 Repo | Access |
|---|---|---|---|
| Public metadata | {org}/security-verifiers-e1-metadata | {org}/security-verifiers-e2-metadata | Public |
| Private canonical | {org}/security-verifiers-e1 | {org}/security-verifiers-e2 | Gated |
Set environment variables in .env:
HF_TOKEN=hf_your_token_here
E1_HF_REPO=your-org/security-verifiers-e1
E2_HF_REPO=your-org/security-verifiers-e2
# Build metadata locally
make hf-e1-meta
make hf-e2-meta
# Push to PUBLIC repos (metadata only)
make hf-e1-push HF_ORG=your-org
make hf-e2-push HF_ORG=your-org
# Push to PRIVATE repos (canonical splits with Features)
make hf-e1p-push-canonical HF_ORG=your-org
make hf-e2p-push-canonical HF_ORG=your-org
# Validate before push
make validate-data
# Push all metadata
make hf-push-all HF_ORG=your-org
Metadata repos provide Dataset Viewer compatibility without exposing sensitive data.
make hf-e1-meta # → build/hf/e1/meta.jsonl
make hf-e2-meta # → build/hf/e2/meta.jsonl
# Default org: intertwine-ai
make hf-e1-push
make hf-e2-push
# Custom org
make hf-e1-push HF_ORG=your-org
make hf-e2-push HF_ORG=your-org
Canonical repos contain full datasets with explicit HuggingFace Features schema.
make validate-e1-data
make validate-e2-data
# or
make validate-data # both
# E1 canonical (train/dev/test splits)
make hf-e1p-push-canonical HF_ORG=your-org
# E2 canonical
make hf-e2p-push-canonical HF_ORG=your-org
Warning: Canonical push uses --force which deletes and recreates the repo. Use only when schema changes are needed.
make hf-e1p-push-canonical-dry HF_ORG=your-org
make hf-e2p-push-canonical-dry HF_ORG=your-org
For users deploying their own Security Verifiers instances:
make data-e1 data-e1-ood
make clone-e2-sources && make data-e2-local
export HF_TOKEN=hf_your_token
export E1_HF_REPO=your-org/security-verifiers-e1-private
export E2_HF_REPO=your-org/security-verifiers-e2-private
make hub-push-datasets
make hub-test-datasets
Private repos use manual gated access to prevent training contamination:
HF_TOKENTemplate READMEs for gated repos are in scripts/hf/templates/.
import os
from datasets import load_dataset
# Set token
os.environ["HF_TOKEN"] = "hf_your_token"
# Load from private repo
dataset = load_dataset(
"your-org/security-verifiers-e1",
split="train",
token=os.environ["HF_TOKEN"]
)
Environments automatically handle dataset loading:
import verifiers as vf
# Auto: tries local → hub → synthetic
env = vf.load_environment("sv-env-network-logs")
# Explicit hub loading
env = vf.load_environment("sv-env-network-logs", dataset_source="hub")
# Synthetic fallback (for testing)
env = vf.load_environment("sv-env-network-logs", dataset_source="synthetic")
401 Unauthorized: Check HF_TOKEN is set and has write access.
Gated access denied: Request access on HF repo page, then set HF_TOKEN.
Schema mismatch: Run make validate-data before push.
Force push warning: Canonical push recreates repos; use only for schema updates.
| Purpose | Location |
|---|---|
| HF push scripts | scripts/hf/ |
| Metadata export | scripts/hf/export_metadata_flat.py |
| Canonical push | scripts/hf/push_canonical_with_features.py |
| Validation scripts | scripts/data/validate_splits_e1.py, validate_splits_e2.py |
| Gated README templates | scripts/hf/templates/ |