scenario-scaffolding

name	scenario-scaffolding
description	Assists with creating complete ITBench scenarios by applying fault mechanisms to specific services, populating scenario files, and generating groundtruth DSL with fault propagations and alert predictions.

Purpose

This skill guides you through the complete scenario creation workflow:

1. Fault Application - Apply fault mechanism to a specific service/component
2. Scenario File Population - Define which application, faults, and tools
3. Ground Truth Generation - Create DSL with groups, propagations, and alerts

When to Use This Skill

This skill auto-activates when:

• Working with files matching **/scenarios/index.json
• Editing files in **/scenarios/files/scenario_*/
• User mentions "scenario scaffold", "ground truth", "disruptions", "DSL"
• Creating or modifying scenario definitions

Prerequisites

Before starting scenario scaffolding:

• ✅ Fault mechanism is fully implemented (Ansible task + fault index complete)
• ✅ You know the target application (prefer OpenTelemetry Demo over BookInfo)
• ✅ You know the specific service/component to target

Workflow

Step 1: Apply Fault to Service/Component

Dynamically identify available services for your chosen application:

1.1 List Available Applications

# Extract all application keys from managers.yaml
grep -E "^  [a-z_]+:" scenarios/sre/roles/applications/defaults/main/managers.yaml | sed 's/://g' | awk '{print $1}'

Application Preference:

• Prefer OpenTelemetry Demo (opentelemetry_demo / Astronomy Shop) for most scenarios - it's richer, more comprehensive, and better maintained
• Use BookInfo (book_info) only if the fault specifically requires its simpler architecture

1.2 Get Application Metadata

# Replace <app-key> with your chosen application
# Get namespace
grep -A 15 "^  <app-key>:" scenarios/sre/roles/applications/defaults/main/managers.yaml | grep "namespace:" | awk '{print $2}'

# Get documentation URL
grep -A 15 "^  <app-key>:" scenarios/sre/roles/applications/defaults/main/managers.yaml | grep -E "url:|documentation:"

1.3 Discover Services from Manifests

# Get the namespace from step 1.2
NAMESPACE="<namespace>"

# Find all Deployments in the application
grep -r "kind: Deployment" scenarios/sre/roles/applications/templates/kubernetes/ | grep "$NAMESPACE" | grep -oP 'name: \K[a-z0-9-]+'

# Find all Services
grep -r "kind: Service" scenarios/sre/roles/applications/templates/kubernetes/ | grep "$NAMESPACE" | grep -oP 'name: \K[a-z0-9-]+'

# Find all StatefulSets
grep -r "kind: StatefulSet" scenarios/sre/roles/applications/templates/kubernetes/ | grep "$NAMESPACE" | grep -oP 'name: \K[a-z0-9-]+'

1.3.1 Optional: Ground in Real Deployment

Ask the user:

Would you like to deploy the application to a live cluster to get actual deployment names, service names, and resource details? This ensures accuracy but requires a running Kubernetes cluster.

If YES:

1. Ask for kubeconfig path:

   What is the path to your kubeconfig file? (e.g., ~/.kube/config)
   

2. Set kubeconfig and deploy (outputs will be shown):

   # Set the kubeconfig
   export KUBECONFIG=<path-from-user>
   
   # Navigate to scenarios directory
   cd scenarios/sre
   
   # Deploy tools - outputs will be displayed
   make deploy-tools
   
   # Deploy applications - outputs will be displayed
   make deploy-applications
   

   Note: Both commands will display their complete output including:

   • Ansible playbook task execution
   • Kubernetes resource creation status
   • Any warnings or errors

3. Query live cluster for actual resource names:

   # Get actual deployments
   kubectl get deployments -n <namespace> -o jsonpath='{.items[*].metadata.name}'
   
   # Get actual services
   kubectl get services -n <namespace> -o jsonpath='{.items[*].metadata.name}'
   
   # Get actual pods (with labels)
   kubectl get pods -n <namespace> --show-labels
   
   # Get actual configmaps
   kubectl get configmaps -n <namespace> -o jsonpath='{.items[*].metadata.name}'
   

4. Use these actual names in your scenario instead of guessing from manifests

If NO: Continue with manifest-based discovery from Step 1.3

1.4 Verify from Documentation

Consult the documentation URL from step 1.2 to understand:

• Service architecture diagram - Shows how services connect and depend on each other
• Component roles and responsibilities
• Dependencies between services (who calls whom)

IMPORTANT: Study the architecture diagram carefully - it's essential for:

• Understanding service dependencies (used in propagations)
• Identifying which downstream services will be affected
• Predicting which alerts will fire based on service relationships

Select the service that best demonstrates the fault mechanism.

Step 2: Populate Scenario Files

The scenario consists of multiple files in scenarios/sre/roles/scenarios/files/scenario_<ID>/:

2.1 Scenario Index Entry

File: scenarios/sre/roles/documentation/files/library/scenarios/index.json

Structure (discovered dynamically):

First, get available tags and platforms:

# Get valid tags
jq '.properties.tags.items.enum' scenarios/sre/roles/documentation/files/library/faults/schema.json

# Get valid platforms
jq '.properties.platforms.items.enum' scenarios/sre/roles/documentation/files/library/faults/schema.json

# Get valid categories
jq '.properties.category.enum' scenarios/sre/roles/documentation/files/library/scenarios/schema.json

Then construct the scenario entry:

{
  "id": <next-available-id>,
  "category": "<category-from-schema>",  // e.g., "sre", "finops", "ciso"
  "complexity": "<complexity>",  // "low", "medium", "high"
  "description": "User-facing incident description",
  "environment": {
    "applications": [
      {
        "id": "<application-id-from-step-1>"  // Prefer "opentelemetry-demo" over "book-info"
      }
    ]
  },
  "platforms": ["<platform-from-schema>"],
  "tags": ["<tag1>", "<tag2>"],  // Match fault tags from schema

  // Populated in this step:
  "disruptions": [
    {
      "injections" 
        
           
           
            
             
               
               
               
                 
                 
              
            
            
          
        
      
      
       
         
          
             
               
          
        
      
    
  

     

   
    
      
         
          
             
             
          
        
      
    
  

2.2 Understanding Disruptions

Discover disruption patterns from existing scenarios:

# Find scenarios using your chosen fault
FAULT_ID="<your-fault-id>"
jq --arg fault "$FAULT_ID" '.[] | select(.disruptions[].injections[].id == $fault) | {id, disruptions}' \
  scenarios/sre/roles/documentation/files/library/scenarios/index.json

# Examine a specific scenario's disruptions
SCENARIO_ID="<scenario-number>"
jq --arg id "$SCENARIO_ID" '.[] | select(.id == ($id | tonumber)) | .disruptions' \
  scenarios/sre/roles/documentation/files/library/scenarios/index.json

Single Fault Injection (template):

{
  "disruptions": [
    {
      "injections": [
        {
          "id": "<fault-id>",
          "args": {
            "kubernetesObject": {
              "apiVersion": "apps/v1",
              "kind": "<Deployment|StatefulSet>",
              "metadata": {
                "name": "<service-name>",
                "namespace": "<namespace>"
              }
            },
            "container": {
              "name": "<container-name>"
            }
          }
        }
      ]
    }
  ]
}

Multiple Fault Injections (template):

{
  "disruptions": [
    {
      "injections": [
        {
          "id": "<fault-id>",
          "args": {
            "kubernetesObject": {
              "apiVersion": "apps/v1",
              "kind": "Deployment",
              "metadata": {"name": "<service-1>", "namespace": "<namespace>"}
            }
          }
        },
        {
          "id": "<fault-id>",
          "args": {
            "kubernetesObject": {
              "apiVersion": "apps/v1"
               
                  
            
          
        
      
    
  

With waitFor Hooks (find examples dynamically):

# Find scenarios using waitFor patterns
jq '.[] | select(.disruptions[].waitFor != null) | {id, disruptions}' \
  scenarios/sre/roles/documentation/files/library/scenarios/index.json | head -50

2.3 waitFor Patterns

When to use waitFor:

• ConfigMap/Secret changes → Restart affected workloads
• Chaos Mesh experiments → Pause/delete schedules
• Multi-step setup → Wait for readiness

Common waitFor IDs:

• restart-kubernetes-workload - Restart a deployment
• wait-kubernetes-workload-ready - Wait for pod readiness

2.4 Solutions Format

Discover solution patterns from the fault definition:

# Get solutions from the fault entry
FAULT_ID="<your-fault-id>"
jq --arg fault "$FAULT_ID" '.[] | select(.id == $fault) | .solutions' \
  scenarios/sre/roles/documentation/files/library/faults/index.json

Adapt fault solutions to scenario context (replace Jinja2 templates with actual values):

Single-step solution (template):

{
  "solutions": [
    [
      {
        "steps": [
          {
            "text": "Revert the last change done to the manifest.",
            "command": "kubectl -n <namespace> rollout undo <kind>/<name>"
          }
        ]
      }
    ],
    [
      {
        "steps": [
          {
            "text": "Manually edit the manifest and fix the issue.",
            "command": "kubectl -n <namespace> edit <kind> <name>"
          }
        ]
      }
    ]
  ]
}

Multi-step solutions (find examples):

# Find scenarios with multi-step solutions
jq '.[] | select(.solutions[][].steps | length > 1) | {id, solutions}' \
  scenarios/sre/roles/documentation/files/library/scenarios/index.json | head -100

2.5 Determine Required Tools

Based on disruptions, identify tools needed (captured by scaffolding from scenarios/sre/roles/scaffolding/tasks/generate_new_scenario_files.yaml):

Chaos Mesh Detection:

enable_chaos_mesh: |-
  {{
    scaffolding_skeleton_scenario.disruptions |
    map(attribute="injections") |
    ansible.builtin.flatten |
    ansible.builtin.selectattr("id", "==", "scheduled-chaos-mesh-experiment") |
    ansible.builtin.length > 0
  }}

This is automatically determined when scenario files are generated.

Step 3: Generate Ground Truth Files

Scenarios require two ground truth files in different formats:

1. groundtruth.yaml - v2 API (simplified entity-based format)
2. groundtruth_v1.yaml - v1 API (DSL format with groups and propagations)

3.0 Create groundtruth.yaml (v2 API)

File: scenarios/sre/roles/scenarios/files/scenario_<ID>/groundtruth.yaml

This is a simplified format that focuses on affected entities and solutions.

Structure:

---
apiVersion: itbench.io/v2
kind: GroundTruth
metadata:
  name: scenario-<ID>
spec:
  # List of expected alerts
  alerts:
    - labels: {}
      name: <alert-name>  # e.g., KubePodNotReady, KubePodCrashLooping

  # List of affected Kubernetes resources (usually root cause)
  entities:
    - apiVersion: <api-version>
      kind: <kind>
      metadata:
        name: <resource-name>
        namespace: <namespace>

  # Solution paths (same as scenario index)
  solutions:
    - - steps:
          - command: <kubectl command>
            text: <step description>

Example (find real examples dynamically):

# View an existing groundtruth.yaml for reference
cat scenarios/sre/roles/scenarios/files/scenario_20/groundtruth.yaml

# Or examine multiple scenarios
ls scenarios/sre/roles/scenarios/files/scenario_*/groundtruth.yaml | head -5 | xargs -I {} sh -c 'echo "=== {} ===" && cat {}'

Template:

---
apiVersion: itbench.io/v2
kind: GroundTruth
metadata:
  name: scenario-<ID>
spec:
  alerts:
    - labels: {}
      name: <alert-name>  # From fault's alerts.application
  entities:
    - apiVersion: apps/v1
      kind: <Deployment|StatefulSet|etc>
      metadata:
        name: <service-name>
        namespace: <namespace>
  solutions:
    - - steps:
          - command: kubectl -n <namespace> rollout undo <kind>/<name>
            text: Revert the last change done to the manifest.
      - steps:
          - command: kubectl -n <namespace> edit <kind> 
                    

Key Points:

• alerts: List primary alerts that will fire (from fault index)
• entities: List the root cause resource(s) being modified by the fault
• solutions: Copy from scenario index (remove Jinja2 templates, use actual values)

3.1 Create groundtruth_v1.yaml (v1 API - DSL Format)

File: scenarios/sre/roles/scenarios/files/scenario_<ID>/groundtruth_v1.yaml

Ground truth uses DSL format (groups) to define fault propagation chains.

3.2 DSL Structure

---
apiVersion: itbench.io/v1
kind: GroundTruth
metadata:
  name: scenario-<ID>
spec:
  # Predicted alerts
  alerts:
    - group_id: <group-id>
      id: <alert-name>
      metadata:
        description: <alert description>

  # Resource groups
  groups:
    - id: <unique-group-id>
      kind: <Pod|Service|Deployment|ConfigMap|etc>
      namespace: <namespace>
      name: <resource-name>  # OR filter: ["regex-pattern"]
      root_cause: true  # At least one group must be root cause

  # Logical relationships
  aliases:
    - [<group-id-1>, <group-id-2>, <group-id-3>]

  # Fault propagations
  propagations:
    - source: <group-id>
      target: <group-id>
      condition: <what causes propagation>
      effect:  

  
  
        
        
      
         
         
         
       

  
  
     
        
            
         

3.3 Defining Groups

Groups represent sets of Kubernetes resources.

Required fields:

• id: Unique identifier
• kind: Kubernetes kind (Pod, Service, Deployment, etc.)
• namespace: Kubernetes namespace
• One of:
  • name: Exact resource name, OR
  • filter: List of regex patterns

Optional fields:

• root_cause: boolean (at least one group must be true)

Discover group patterns from existing scenarios:

# View groups from a specific scenario
cat scenarios/sre/roles/scenarios/files/scenario_<ID>/groundtruth_v1.yaml | grep -A 10 "^  groups:"

# Find scenarios with ConfigMap root causes
grep -r "kind: ConfigMap" scenarios/sre/roles/scenarios/files/*/groundtruth_v1.yaml

Pod group with filter (template):

groups:
  - id: <service-name>-pod-1
    kind: Pod
    namespace: <namespace>
    filter:
      - <service-name>-.*
    root_cause: true

Service group with filter (template):

groups:
  - id: <service-name>-service-1
    kind: Service
    namespace: <namespace>
    filter:
      - <service-name>\b  # \b = word boundary

ConfigMap group (root cause) (template):

groups:
  - id: <config-name>-cm
    kind: ConfigMap
    namespace: <namespace>
    name: <configmap-name>
    root_cause: true

3.4 Defining Aliases

Aliases link related groups logically.

aliases:
  - [<group-id-1>, <group-id-2>, <group-id-3>]

Discover alias patterns:

# Find scenarios with aliases
grep -A 5 "^  aliases:" scenarios/sre/roles/scenarios/files/*/groundtruth_v1.yaml | head -20

Template:

aliases:
  - - <service-name>-pod-1
    - <service-name>-service-1
  - - <dependent-service>-pod-1
    - <dependent-service>-service-1

3.5 Defining Propagations

Propagations describe how faults spread through the system.

IMPORTANT: Use the architecture diagram from the application's documentation to:

1. Identify which services depend on the affected service
2. Map the propagation path (root cause → affected services → downstream services)
3. Understand the call chain and service relationships

Required fields:

• source: Group ID where propagation starts
• target: Group ID where propagation ends
• condition: What causes the propagation (based on architecture)
• effect: What observable impact results

Discover propagation patterns:

# Find propagation examples
grep -A 10 "^  propagations:" scenarios/sre/roles/scenarios/files/*/groundtruth_v1.yaml | head -50

Template:

propagations:
  - source: <root-cause-group-id>
    target: <affected-service-group-id>
    condition: <what triggers propagation>
    effect: <observable impact>
  - source: <affected-service-group-id>
    target: <dependent-service-group-id>
    condition: <dependency relationship>
    effect: <downstream impact>

3.6 Predicting Alerts

IMPORTANT: Always read alert definitions from actual source files to get up-to-date alert rules.

Use the Architecture Diagram:

• Review the application's architecture diagram from documentation
• Identify which services are affected by the fault (directly and indirectly)
• Map fault symptoms to services in the diagram to predict which alerts will fire

Alert Sources:

1. Application-Specific Alerts - Read from local PrometheusRules:

   OpenTelemetry Demo:

   cat scenarios/sre/roles/applications/templates/kubernetes/otel_demo/prometheusrules.j2
   

   BookInfo:

   cat scenarios/sre/roles/applications/templates/kubernetes/book_info/prometheusrules.j2
   

2. Kubernetes Platform Alerts - Check:

   a. Local schema (available alerts in ITBench):

   jq '.properties.alerts.properties.application.items.enum' \
     scenarios/sre/roles/documentation/files/library/faults/schema.json
   

   b. Prometheus Community Rules (canonical source):

   • Reference: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/templates/prometheus/rules-1.14
   • Search for alerts matching your fault symptoms (e.g., KubePodCrashLooping, KubePodNotReady)

How to predict alerts:

1. Read PrometheusRules files for the target application
2. Match fault symptoms to alert expressions
3. Identify which alerts will trigger based on fault behavior

Alert format (template):

alerts:
  - group_id: <affected-service-group-id>
    id: <alert-name-from-prometheus-rules>
    metadata:
      description: <alert description matching PrometheusRules>
  - group_id: <affected-service-group-id>
    id: <another-alert-name>
    metadata:
      description: <another alert description>

3.7 Complete Example (groundtruth_v1.yaml)

View real examples dynamically:

# View a complete groundtruth_v1.yaml file
cat scenarios/sre/roles/scenarios/files/scenario_20/groundtruth_v1.yaml

# Compare multiple scenarios for patterns
for scenario in 20 30 40; do
  echo "=== Scenario $scenario ==="
  cat "scenarios/sre/roles/scenarios/files/scenario_${scenario}/groundtruth_v1.yaml" 2>/dev/null || echo "Not found"
  echo ""
done

Complete Template:

---
apiVersion: itbench.io/v1
kind: GroundTruth
metadata:
  name: scenario-<ID>
spec:
  alerts:
    - group_id: <affected-service-group-id>
      id: <alert-name>
      metadata:
        description: <alert description from PrometheusRules>

  groups:
    - id: <root-cause-group-id>
      kind: <Pod|Deployment|ConfigMap>
      namespace: <namespace>
      filter:
        - <name-pattern-regex>
      root_cause: true

    - id: <affected-service-group-id>
      kind: Service
      namespace: <namespace>
      filter:
        - <service-name>\b

    - id: <dependent-service-group-id>
      kind: Service
      namespace: <namespace>
      filter:
        - <dependent-service-name>\b

  
      
       

  
      
       
        
        
      
       
        
        

  
      
        
      
         
         
         
       

  
     
        
            
         
     
        
            
         

Validation & Generation

IMPORTANT: You must manually create both groundtruth files before running validation:

1. Create groundtruth.yaml (v2 API - simplified format)
2. Create groundtruth_v1.yaml (v1 API - DSL format)

After creating both files manually, validate with:

cd scenarios/sre
make regenerate-scenario-files

This validates and generates:

• scenario_<ID>/scenario.yaml - Scenario spec (auto-generated from index)

Both groundtruth files must be created manually - they are not auto-generated.

Common Patterns

Discover patterns dynamically by analyzing existing scenarios:

Identify Application-Specific Patterns

# Get all scenarios for a specific application
APP_ID="opentelemetry-demo"  # Prefer opentelemetry-demo over book-info
jq --arg app "$APP_ID" '.[] | select(.environment.applications[].id == $app) | {id, description}' \
  scenarios/sre/roles/documentation/files/library/scenarios/index.json

# Analyze groundtruth patterns for that application
for scenario_id in $(jq --arg app "$APP_ID" '.[] | select(.environment.applications[].id == $app) | .id' \
  scenarios/sre/roles/documentation/files/library/scenarios/index.json); do
  echo "=== Scenario $scenario_id ==="
  cat "scenarios/sre/roles/scenarios/files/scenario_${scenario_id}/groundtruth_v1.yaml" | grep -E "^  (groups|propagations):" -A 20
done

Common Propagation Patterns

Find typical propagation chains:

# Find Pod → Service propagations
grep -A 4 "source:.*pod" scenarios/sre/roles/scenarios/files/*/groundtruth_v1.yaml | grep "target:" | head -10

# Find Service → Service propagations
grep -A 4 "source:.*service" scenarios/sre/roles/scenarios/files/*/groundtruth_v1.yaml | grep "target:" | head -10

Generic propagation chain template:

<Root Cause Resource> (root_cause) → <Affected Service> → <Dependent Service> → Alerts

ConfigMap Fault Patterns

Discover ConfigMap scenarios:

# Find scenarios with ConfigMap root causes
grep -r "kind: ConfigMap" scenarios/sre/roles/scenarios/files/*/groundtruth_v1.yaml -l | \
  xargs -I {} sh -c 'echo "=== {} ===" && cat {} | head -50'

Template:

groups:
  - id: <config-name>-cm
    kind: ConfigMap
    namespace: <namespace>
    name: <configmap-name>
    root_cause: true

  - id: <affected-workload>-pod
    kind: Pod
    namespace: <namespace>
    filter: [<workload-name>-.*]

propagations:
  - source: <config-name>-cm
    target: <affected-workload>-pod
    condition: ConfigMap contains <type of issue>
    effect: <workload> pod <impact>

Anti-Patterns

❌ Don't:

• Forget to set root_cause: true on at least one group
• Use duplicate group IDs
• Create propagations with undefined group IDs
• Skip alert prediction
• Use entities format (use groups instead)
• Leave faultId in final scenario (it's a scaffolding hint)
• Use obvious fault-revealing names for injected resources (e.g., malformed-config, fault-injector, crash-trigger, chaos-configmap)

✅ Do:

• Mark the actual fault injection point as root cause
• Create groups for all affected resources
• Define propagations showing fault spread
• Predict alerts based on PrometheusRules
• Use regex filters for pod groups
• Test scenario generation with make regenerate-scenario-files
• Remove faultId after using it to build disruptions
• Use non-obvious names for injected resources (e.g., app-config, recommendation-features, sidecar-processor, cache-helper) - Rationale: Agents should diagnose issues based on symptoms and observability, not by discovering obviously-named fault injection resources

Reference Examples

Discover reference scenarios dynamically by complexity:

# Find simple scenarios (low complexity)
jq '.[] | select(.complexity == "low") | {id, description, faults: [.disruptions[].injections[].id]}' \
  scenarios/sre/roles/documentation/files/library/scenarios/index.json | head -50

# Find complex scenarios (high complexity)
jq '.[] | select(.complexity == "high") | {id, description, faults: [.disruptions[].injections[].id]}' \
  scenarios/sre/roles/documentation/files/library/scenarios/index.json | head -50

# Find scenarios using specific fault mechanisms
FAULT_ID="<your-fault-id>"
jq --arg fault "$FAULT_ID" '.[] | select(.disruptions[].injections[].id == $fault) | {id, description}' \
  scenarios/sre/roles/documentation/files/library/scenarios/index.json

View groundtruth files for reference:

# List all available groundtruth files
ls scenarios/sre/roles/scenarios/files/scenario_*/groundtruth_v1.yaml | sort -V

# View specific scenarios
cat scenarios/sre/roles/scenarios/files/scenario_1/groundtruth_v1.yaml   # Feature flag pattern
cat scenarios/sre/roles/scenarios/files/scenario_20/groundtruth_v1.yaml  # Image pull error
cat scenarios/sre/roles/scenarios/files/scenario_40/groundtruth_v1.yaml  # Code change pattern

Tips

1. Start with the architecture diagram - Review the application's architecture diagram from documentation to understand service relationships
2. Identify the fault mechanism - What resource is being modified?
3. Identify the root cause group - The resource directly affected by the fault
4. Map dependencies using the diagram - Follow the architecture to see which services call the affected service
5. Predict observables - Use the diagram to identify which services will be impacted and what alerts will fire
6. Define propagation chain - Trace the fault spread through the architecture (root cause → dependencies → downstream effects)
7. Test generation - Always run make regenerate-scenario-files to validate

Next Steps

After scenario scaffolding:

1. Test the complete scenario in a development cluster
2. Verify alerts fire as predicted
3. Validate solutions actually work
4. Commit changes: git add . && git commit -m "feat: add scenario <ID>"