runbooks-manager

name	runbooks-manager
description	Build, match, and manage security investigation runbooks. Create from analyst investigations or SOAR playbooks, match alerts to existing runbooks, or compose new ones from patterns. Self-contained skill for all runbook operations.

Runbooks Manager Skill

Overview

This skill handles all runbook operations for the Analysi Security platform:

• Building: Convert analyst investigations, SOAR playbooks, or detection documentation into standardized runbook.md files
• Matching: Find the most relevant investigation runbook for incoming OCSF alerts
• Composing: Intelligently blend patterns from existing runbooks when exact matches don't exist

Runbooks define hypothesis-driven investigation procedures that specify WHAT information is needed and WHY (not HOW - implementation is handled by alert-planner).

CRITICAL: Runbook Scope & Definition of Done

The Fundamental Principle

Runbooks are for DETECTION RULES, not individual alerts.

When you receive an example alert to build or match a runbook, that alert is just ONE INSTANCE. The runbook you create or select must be generic enough to handle ALL alerts generated by the same detection rule.

Why This Matters

A detection rule identifies specific TTPs (Tactics, Techniques, and Procedures). All alerts from that rule share:

• The same attack patterns being detected
• Similar evidence sources and investigation steps
• Common hypotheses to validate
• Consistent decision criteria

The example alert informs but does not define the runbook.

Definition of Done

A runbook is complete when it:

1. ✅ Handles the current alert - The example case can be investigated using this runbook
2. ✅ Handles ALL instances - Any future alert from the same detection rule can use this runbook
3. ✅ Covers the full outcome space - Supports both TP and FP conclusions with appropriate investigation paths
4. ✅ Doesn't over-fit to the example - Even if the example is an obvious FP, the runbook still includes full investigation steps for potential TPs

Anti-Pattern: Over-fitting to Examples

❌ WRONG: Example alert shows obvious scanner activity → Create runbook that only checks for scanners

✅ RIGHT: Example alert shows obvious scanner activity → Create runbook that:

• Validates scanner hypothesis (which will confirm the example case)
• BUT ALSO validates targeted attack hypothesis (for future non-scanner cases)
• AND validates successful exploitation hypothesis (for future breach cases)

Practical Guidance

When building or composing a runbook:

1. Use the example to understand the detection - What is this rule trying to catch?
2. Think about the rule's full scope - What are ALL the scenarios this rule might fire on?
3. Build for the rule, validate with the example - The example is a test case, not the spec
4. Include hypotheses for both benign and malicious - Cover FPs and TPs equally

Prerequisites

This skill assumes you understand:

• task-builder: Runbooks will be converted to Tasks
• workflow-builder: Runbooks map to Workflow compositions
• cy-language-programming: Output patterns match Cy capabilities
• OCSF Schema: Use mcp__cy-script-assistant__get_ocsf_schema for field definitions

Two Primary Modes

Mode 1: Building Runbooks

Use when:

• Creating a new runbook for a detection rule
• Converting analyst investigation notes to standardized format
• Migrating SOAR playbooks to runbook.md format
• Building reusable investigation patterns (sub-runbooks)

See: references/building/ directory for complete building guidance

Mode 2: Matching & Composition

Workflow Decision Tree:

When presented with an OCSF alert:

1. Exact detection_rule match? → Return that runbook ("matched", VERY HIGH confidence)
2. No exact match → Compose a new runbook:
   • Score existing runbooks to rank candidates
   • Assemble from common/ building blocks: universal/ + by_source/ + by_type/ + evidence/
   • Draw conditional logic and decision points from top-scoring candidates
   • Confidence: HIGH (strong by_type/ coverage) → VERY LOW (no relevant patterns)

See: references/matching/ directory for matching and composition guidance

CRITICAL: Step Criticality Markers (★)

THE MOST IMPORTANT ASPECT: Clearly identify which steps are critical for minimum viable investigation.

★ = Must Execute

• Critical steps are marked with ★
• These steps MUST run for the investigation to be valid
• Without these, you cannot reach a defensible verdict
• Typically 3-5 steps per runbook

No Marker = Optional/Degradable

• Enrichment steps that add context but aren't essential
• Can skip if integration unavailable or for speed
• Investigation remains valid without these

How to Identify Critical Steps:

1. Consensus across analysts - All analysts did it → ★
2. Required for verdict - Can't determine TP/FP without it → ★
3. Core evidence - Primary data about the attack → ★
4. Optional enrichment - Nice to have → No marker

Example:

### 1. Alert Understanding ★         # CRITICAL: Must understand what we're investigating
### 2a. SIEM Evidence Collection ★    # CRITICAL: Core evidence from our environment
### 2b. IP Reputation Check           # OPTIONAL: External enrichment
### 3. Attack Success Determination ★ # CRITICAL: Can't reach verdict without this
### 4. Final Analysis ★               # CRITICAL: Required output for SOC

Critical: Runbook vs Implementation Distinction

Runbooks are investigation guides, NOT implementation specifications.

Runbook (This Skill)	Implementation (Alert-Planner)
Suggests "check IP reputation"	Decides to use VirusTotal vs Cisco Talos
Says "search SIEM for related events"	Chooses Splunk vs SentinelOne
Specifies "decode payload patterns"	Selects specific decoding tools
Indicates "threat intelligence enrichment"	Picks available TI integrations
Uses get_src_ip(alert) (OCSF helper)	Resolves evidences[].src_endpoint.ip at runtime

Key Principle: Runbooks describe WHAT information is needed and WHY, while the planner determines HOW to get it based on available tools.

Pattern to Cy Mapping

How runbook patterns translate to Cy/Task implementation:

Runbook Pattern	Cy Tool	Task Function	Description
hypothesis_formation	llm_run	reasoning	Form investigation theories
evidence_correlation	llm_run	reasoning	Correlate evidence sources
payload_analysis	llm_run	enrichment	Decode and analyze payloads
impact_assessment	llm_run	reasoning	Determine attack success
threat_synthesis	llm_run	synthesis	Full context analysis
integration_query	app::integration::action	search/enrichment	Direct integration calls

Match Scoring Overview

The skill uses a weighted scoring system with intentionally documented priorities:

WEIGHTS = {
    'exact_detection_rule': 100,    # Exact match on detection rule name
    'subcategory_match': 40,        # Same subcategory (e.g., SQL Injection = SQL Injection)
    'subcategory_similar': 25,      # Similar subcategory family (e.g., SQL Injection ~ Command Injection)
    'alert_type_match': 20,         # Same broad alert type (e.g., Web Attack = Web Attack)
    'source_category': 30,          # Same source (WAF, EDR, etc.)
    'mitre_overlap': 20,            # Per overlapping MITRE tactic
    'integration_compatibility': 15, # Has required integrations available
    'cve_same_vendor': 35,          # Same vendor for CVE attacks
    'cve_same_year': 10,            # Same year for CVE attacks
}

Note: subcategory_match (40) is the primary attack classifier — it matches specific attack types like "SQL Injection", "XSS", "LFI". alert_type_match (20) is the broad category like "Web Attack" or "Brute Force". These are separate dimensions. See scripts/match_scorer.py for the implementation.

Decision Boundary: Exact detection_rule string match = "matched". Everything else = "composed". Scores rank candidates for composition, not for declaring a match.

Confidence Levels:

• VERY HIGH: Exact detection_rule match + comprehensive runbook — use as-is
• HIGH: Exact match (sparse) OR composed with strong by_type/ pattern coverage
• MEDIUM: Composed from multiple relevant by_type/ patterns — review recommended
• LOW: Composed with limited pattern coverage — significant review needed
• VERY LOW: No relevant patterns — composed from universal blocks only

See: references/matching/confidence-rubric.md for complete rubric

Repository Structure

Self-contained within this skill:

.claude/skills/runbooks-manager/
├── repository/         # 14+ production runbooks (use these as examples!)
├── common/             # Reusable sub-runbook components
│   ├── universal/      # Always applicable steps
│   ├── by_source/      # Source-specific patterns (WAF, EDR)
│   ├── by_type/        # Attack type patterns
│   └── evidence/       # Evidence collection patterns
├── templates/          # Template for creating new runbooks
│   └── runbook-template.md
├── scripts/
│   ├── build_runbook_index.py  # Generate runbook index (JIT)
│   ├── match_scorer.py         # Matching algorithm reference
│   └── validate_runbook.py     # Runbook structure validation
├── references/
│   ├── building/       # Complete building guidance
│   ├── matching/       # Complete matching guidance
│   └── shared/         # Common documentation
└── examples/
    └── matching/       # Alert matching scenario examples

Note: Index files are generated on-demand (JIT) in a temp directory. See "Just-In-Time Indexing" section below.

Just-In-Time Indexing

Index files are generated on-demand rather than pre-committed to the repository. This ensures indexes are always fresh and eliminates sync issues.

Generating the Index

# Generate index in auto-created temp directory
# Captures the path via INDEX_DIR=<path> output
INDEX_DIR=$(python3 scripts/build_runbook_index.py | grep INDEX_DIR | cut -d= -f2)

# Or specify a custom output directory
python3 scripts/build_runbook_index.py --output-dir /tmp/my-runbook-index

Generated Index Structure

The script creates these files in the output directory:

$INDEX_DIR/
├── all_runbooks.json           # Complete metadata for all runbooks
├── master_index.json           # Summary with categories and counts
├── by_subcategory/             # Index by attack subcategory
│   ├── sql-injection.md
│   ├── xss.md
│   └── ...
├── by_attack_type/             # Index by broad attack type
├── by_source_category/         # Index by alert source (WAF, EDR)
├── by_mitre_tactic/            # Index by MITRE ATT&CK tactic
├── by_integration/             # Index by required integrations
├── by_vendor/                  # Index by vendor (for CVE runbooks)
└── by_cve_year/                # Index by CVE year

Using the Index

1. Read all_runbooks.json for complete runbook metadata
2. Read dimensional indexes (e.g., by_subcategory/sql-injection.md) to find candidates
3. Apply scoring algorithm from scripts/match_scorer.py
4. Interpret confidence using references/matching/confidence-rubric.md

Cleanup

After matching is complete, remove the temp directory:

rm -rf $INDEX_DIR

Best Practices

DO:

• ALWAYS mark critical steps with ★ (this is the MOST important)
• Identify 3-5 critical steps per runbook (minimum viable investigation)
• Use analyst consensus to determine criticality
• Use WikiLinks (![[...]]) for common patterns
• Use hypothesis-driven investigation structure
• Reference OCSF alert fields using direct paths or helper functions (see references/shared/ocsf-field-reference.md)
• Generate fresh index before matching (JIT indexing)
• Copy existing patterns when composing (don't invent new ones)
• Document provenance when composing runbooks
• Clean up temp index directory after matching completes

DON'T:

• Include severity in runbook metadata (it's alert-specific)
• Create duplicate patterns (use sub-runbooks)
• Mix runbook logic with task creation (separate concerns)
• Fabricate queries that seem logical but weren't actually used by analysts
• Skip validation before using runbooks

References

Building runbooks:

• references/building/format-specification.md - Complete YAML and markdown format
• references/building/creating-runbooks.md - Step-by-step creation guide
• references/building/sub-runbook-patterns.md - When and how to extract patterns
• references/building/quality-guide.md - Quality checklist and validation
• references/building/data-flow-specification.md - OCSF to enrichments flow

Matching runbooks:

• references/matching/matching-algorithm.md - Scoring weights and criteria
• references/matching/confidence-rubric.md - Complete confidence system
• references/matching/composition-guide.md - Composition strategies

Shared:

• references/shared/ocsf-field-reference.md - OCSF field paths and helper function reference
• references/shared/repository-structure.md - Repository organization details

Templates & Examples

Template:

• templates/runbook-template.md - Skeleton template for new runbooks

Production Runbook Examples (use these as reference):

• repository/sql-injection-detection.md - Complete web attack example with WikiLinks
• repository/command-injection-detection.md - Command injection with SIEM queries
• repository/xss-detection.md - XSS detection and analysis
• repository/palo-alto-pan-os-command-injection-cve-2024-3400.md - CVE-based investigation
• All 14 runbooks in repository/ are production-quality examples

Matching Examples:

• examples/matching/ - Alert matching scenario examples