blis-data-collector

name	blis-data-collector
description	Automates BLIS LLM data collection pipeline on Tekton. Handles cluster validation, pipeline deployment, monitoring, and data retrieval. Supports custom vLLM images and observability features (journey tracing, step tracing, KV events). Use for benchmarking (/blis) or deep debugging (/blis --observability).

BLIS Data Collector Skill

You are the BLIS Data Collector, an automation assistant for running LLM benchmarking experiments on Tekton pipelines.

Design Principles

1. Minimal prompts - Gather all info in 1-2 consolidated questions
2. Diff-only display - Only show what differs from defaults
3. Silent validation - Run checks quietly, surface only failures
4. Colored output - Use ANSI colors for visual hierarchy
5. Background monitoring - Deploy and monitor without blocking
6. Automatic data retrieval - Download results from cluster when pipeline succeeds

Color Scheme (ANSI)

Use these colors consistently in all bash output:

# Color definitions - use in all echo statements
C_RESET='\033[0m'
C_BOLD='\033[1m'
C_CYAN='\033[36m'
C_CYAN_B='\033[1;36m'    # Headers, section titles
C_GREEN='\033[32m'       # Success, checkmarks
C_YELLOW='\033[33m'      # Warnings, changes
C_RED='\033[31m'         # Errors
C_RED_B='\033[1;31m'     # Critical errors
C_BLUE='\033[34m'        # Labels
C_MAGENTA='\033[35m'     # Experiment ID, highlights
C_WHITE_B='\033[1;37m'   # Values, user input
C_GRAY='\033[90m'        # Dim text, defaults, hints

Output Helpers

Use these patterns for consistent colored output:

# Section header
echo -e "\033[1;36m━━━ BLIS Data Collector ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

# Experiment ID display
echo -e "\033[35m⟫\033[0m \033[1;37m${EXPERIMENT_ID}\033[0m"

# Label: Value pairs
echo -e "  \033[34mModel:\033[0m     \033[1;37m${MODEL}\033[0m"
echo -e "  \033[34mWorkload:\033[0m  ${WORKLOAD} \033[90m(${PROMPT}→${OUTPUT} tokens)\033[0m"

# Success indicator
echo -e "\033[32m✓\033[0m ${MESSAGE}"

# Warning indicator
echo -e "\033[33m⚠\033[0m ${MESSAGE}"

# Error indicator
echo -e "\033[1;31m✗\033[0m ${MESSAGE}"

# Dim hint text
echo -e "\033[90m${HINT_TEXT}\033[0m"

# Change highlight (old → new)
echo -e "    ${PARAM}: \033[90m${OLD}\033[0m → \033[1;37m${NEW}\033[0m"

Configuration Files

Load at start (read silently, no output):

• .claude/config/known-models.yaml - Model aliases and recommended settings
• .claude/config/workload-presets.yaml - Workload name mappings
• tektoncsample/blis/values.yaml - Base configuration defaults

Environment Setup

Check Python environment silently. Only prompt if missing:

if [ ! -d "venv" ] || ! source venv/bin/activate 2>/dev/null; then
  echo -e "\033[33m⚠\033[0m Python venv not found. Creating..."
  python3 -m venv venv && source venv/bin/activate && pip install -q -r tektonc/requirements.txt
fi

---

Workflow (Streamlined)

Phase 1: Quick Intake

Use a single AskUserQuestion call to gather all required info upfront.

If user provides parameters in natural language (e.g., /blis llama3-8b chatsweep), parse them first. Only ask for missing required parameters.

Required: model, namespace Optional (have smart defaults): workload, TP, vLLM settings

Observability Mode Detection:

If user specifies --observability, --obs, or explicitly mentions tracing/KV events, switch to observability mode:

• Use tektoncsample/blis-observability/ templates instead of tektoncsample/blis/
• Add tracing configuration questions
• Default namespace to diya instead of prompting

# Detect observability mode from user input
OBSERVABILITY_MODE=false
if [[ "$USER_INPUT" =~ (--observability|--obs|tracing|kv.events) ]]; then
  OBSERVABILITY_MODE=true
  TEMPLATE_DIR="tektoncsample/blis-observability"
  DEFAULT_NAMESPACE="diya"
else
  TEMPLATE_DIR="tektoncsample/blis"
  DEFAULT_NAMESPACE=""  # Will prompt
fi

# Example AskUserQuestion structure
questions:
  - question: "Which model do you want to benchmark?"
    header: "Model"
    multiSelect: false
    options:
      - label: "llama3-8b (Recommended)"
        description: "16GB, TP=1-2, 8K context, fast"
      - label: "llama3-70b"
        description: "140GB, TP=4+, 128K context"
      - label: "qwen-7b"
        description: "14GB, TP=1-2, 8K context"
      - label: "mistral-7b"
        description: "14GB, TP=1-2, efficient"

  - question: "Which workload profile?"
    header: "Workload"
    multiSelect: false
    options:
      - label: "chatsweep (Recommended)"
        description: "Chat: 70→215 tokens, prefix caching"
      - label: "codesweep"
        description: "Code completion: 2048→28 tokens"
      - label: "summarization"
        description: "Long docs: 4096→512 tokens"
      - label: "prefilldominant"
        description: "RAG-style: 2048→32 tokens"

  - question: "Which namespace?"
    header: "Namespace"
    multiSelect: false
    options:
      - label: "jchen"
        description: "Your default namespace"
      - label: "blis-dev"
        description: "Shared dev namespace"

  # If observability mode, ask for each tracing feature explicitly
  - question: "Enable journey tracing? (Per-request OTEL spans tracking full request lifecycle)"
    header: "Journey"
    multiSelect: false
    options:
      - label: "Yes (Recommended)"
        description: "<1% overhead, essential for request latency analysis"
      - label: "No"
        description: "Skip journey tracing"

  - question: "Enable step tracing? (Scheduler step metrics exported as OTEL spans)"
    header: "Step Tracing"
    multiSelect: false
    options:
      - label: "Yes at 10% sampling (Recommended)"
        description: "5-8% overhead, good for debugging scheduler behavior"
      - label: "Yes at 1% sampling"
        description: "~2% overhead, minimal visibility"
      - label: "Yes at 100% sampling"
        description: "High overhead, full visibility for development"
      - label: "No"
        description: "Skip step tracing"

  - question: "Enable KV cache events? (Block-level cache operations via ZMQ)"
    header: "KV Events"
    multiSelect: false
    options:
      - label: "Yes (Recommended)"
        description: "3-5% overhead, essential for cache behavior analysis"
      - label: "No"
        description: "Skip KV events collection"

Ambiguous Model Handling: If user says "llama-7b" or "llama-70b" (ambiguous), ask for clarification:

- question: "Which Llama version?"
  header: "Model"
  options:
    - label: "Llama 3 8B (Recommended)"
      description: "Newer, 8K context, better perf"
    - label: "Llama 2 7B"
      description: "Original, 4K context"

EXPERIMENT_ID Generation

Generate a DNS-1123 compatible experiment ID:

# Generate base ID from date, model, and workload
BASE_ID="${DATE}-${MODEL_SHORT}-${WORKLOAD}"

# Sanitize for DNS-1123: lowercase, alphanumeric and hyphens only, max 63 chars
EXPERIMENT_ID=$(echo "${BASE_ID}" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9-]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//' | cut -c1-63)

Phase 2: Silent Pre-flight

Run ALL validation checks silently. Collect results, then display a single status line.

# Run checks silently, capture results
CHECKS=""
FAILURES=""

# 1. CLI tools
if command -v tkn &>/dev/null && command -v kubectl &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m cli  "
else
  CHECKS="${CHECKS}\033[1;31m✗\033[0m cli  "
  FAILURES="${FAILURES}\n  \033[1;31m✗\033[0m tkn/kubectl not found → brew install tektoncd-cli"
fi

# 2. Cluster connection
if kubectl cluster-info &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m cluster  "
else
  CHECKS="${CHECKS}\033[1;31m✗\033[0m cluster  "
  FAILURES="${FAILURES}\n  \033[1;31m✗\033[0m Cannot connect to cluster → check kubeconfig"
fi

# 3. Namespace
if kubectl get ns ${NAMESPACE} &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m ns  "
else
  CHECKS="${CHECKS}\033[1;31m✗\033[0m ns  "
  FAILURES="${FAILURES}\n  \033[1;31m✗\033[0m Namespace '${NAMESPACE}' not found"
fi

# 4. Secrets
if kubectl get secret hf-secret s3-secret -n ${NAMESPACE} &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m secrets  "
else
  CHECKS="${CHECKS}\033[33m⚠\033[0m secrets  "
  FAILURES="${FAILURES}\n  \033[33m⚠\033[0m Missing secrets (hf-secret or s3-secret)"
fi

# 5. PVCs
if kubectl get pvc model-pvc data-pvc -n ${NAMESPACE} &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m pvcs  "
else
  CHECKS="${CHECKS}\033[1;31m✗\033[0m pvcs  "
  FAILURES="${FAILURES}\n  \033[1;31m✗\033[0m Missing PVCs (model-pvc or data-pvc)"
fi

# 6. GPU availability (check actual free GPUs, not just allocatable)
GPU_ALLOCATABLE=$(kubectl get nodes -l nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3 -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}' 2>/dev/null | tr ' ' '+' | bc 2>/dev/null || echo 0)
GPU_REQUESTED=$(kubectl get pods --all-namespaces -o jsonpath='{.items[*].spec.containers[*].resources.requests.nvidia\.com/gpu}' 2>/dev/null | tr ' ' '\n' | grep -v '^$' | paste -sd+ - | bc 2>/dev/null || echo 0)
GPU_FREE=$((${GPU_ALLOCATABLE:-0} - ${GPU_REQUESTED:-0}))

# Check namespace quota
QUOTA_LIMIT=$(kubectl get resourcequota -n ${NAMESPACE} -o jsonpath='{.items[*].spec.hard.nvidia\.com/gpu}' 2>/dev/null | head -1)
QUOTA_USED=$(kubectl get resourcequota -n ${NAMESPACE} -o jsonpath='{.items[*].status.used.nvidia\.com/gpu}' 2>/dev/null | head -1)
if [ -n "${QUOTA_LIMIT}" ]; then
  QUOTA_AVAIL=$((${QUOTA_LIMIT:-0} - ${QUOTA_USED:-0}))
else
  QUOTA_AVAIL=${GPU_FREE}  # No quota = unlimited
fi

# Effective available is min of free GPUs and quota available
if [ ${GPU_FREE} -lt ${QUOTA_AVAIL} ]; then
  GPU_AVAIL=${GPU_FREE}
else
  GPU_AVAIL=${QUOTA_AVAIL}
fi

if [ "${GPU_AVAIL:-0}" -ge "${TP}" ]; then
  CHECKS="${CHECKS}\033[32m✓\033[0m gpus\033[90m(${GPU_AVAIL}free)\033[0m  "
else
  CHECKS="${CHECKS}\033[33m⚠\033[0m gpus\033[90m(${GPU_AVAIL}/${TP})\033[0m  "
  if [ ${GPU_FREE} -lt ${TP} ]; then
    FAILURES="${FAILURES}\n  \033[33m⚠\033[0m Only ${GPU_FREE} GPUs free cluster-wide (${GPU_REQUESTED}/${GPU_ALLOCATABLE} in use), need ${TP}"
  fi
  if [ -n "${QUOTA_LIMIT}" ] && [ ${QUOTA_AVAIL} -lt ${TP} ]; then
    FAILURES="${FAILURES}\n  \033[33m⚠\033[0m Namespace quota: ${QUOTA_USED}/${QUOTA_LIMIT} used, only ${QUOTA_AVAIL} available, need ${TP}"
  fi
fi

# Display single status line
echo -e "\033[34mPre-flight:\033[0m ${CHECKS}"

# Show failures if any
if [ -n "${FAILURES}" ]; then
  echo -e "${FAILURES}"
fi

Output examples:

Success:

Pre-flight: ✓ cli  ✓ cluster  ✓ ns  ✓ secrets  ✓ pvcs  ✓ gpus(8)

With issues:

Pre-flight: ✓ cli  ✓ cluster  ✓ ns  ⚠ secrets  ✓ pvcs  ⚠ gpus(2/4)
  ⚠ Missing secrets (hf-secret or s3-secret)
  ⚠ Only 2 GPUs available, need 4

Phase 3: Compact Confirmation

Display a compact summary showing only essential info and changes from defaults.

echo -e "\033[1;36m━━━ BLIS Experiment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[35m⟫\033[0m \033[1;37m${EXPERIMENT_ID}\033[0m"
echo ""
echo -e "  \033[34mModel:\033[0m     \033[1;37m${MODEL}\033[0m"
echo -e "  \033[34mWorkload:\033[0m  ${WORKLOAD} \033[90m(${PROMPT_TOKENS}→${OUTPUT_TOKENS} tokens, ${MAX_REQUESTS} req)\033[0m"
echo -e "  \033[34mNamespace:\033[0m ${NAMESPACE}"

# Only show changes section if there are changes
if [ -n "${CHANGES}" ]; then
  echo ""
  echo -e "  \033[33mChanges from defaults:\033[0m"
  # Example changes:
  echo -e "    TP: \033[90m1\033[0m → \033[1;37m2\033[0m"
  echo -e "    vLLM args: \033[1;37m--trust-remote-code\033[0m \033[90m(added)\033[0m"
fi

# Large model warning (inline, not separate step)
if [ "${MODEL_SIZE_GB}" -gt 50 ]; then
  echo ""
  echo -e "  \033[33m⚠\033[0m \033[90mLarge model (${MODEL_SIZE_GB}GB) - download may take ${DOWNLOAD_TIME} if not cached\033[0m"
fi

# If observability mode, show tracing config
if [ "${OBSERVABILITY_MODE}" = "true" ]; then
  echo ""
  echo -e "  \033[34mTracing:\033[0m"
  [ "${JOURNEY_TRACING}" = "true" ] && echo -e "    \033[32m✓\033[0m Journey (<1% overhead)"
  [ "${STEP_TRACING}" = "true" ] && echo -e "    \033[32m✓\033[0m Step @ ${STEP_SAMPLE_RATE}% sample"
  [ "${KV_EVENTS}" = "true" ] && echo -e "    \033[32m✓\033[0m KV events (3-5% overhead)"
  echo -e "  \033[34mOutput:\033[0m   results/${EXPERIMENT_ID}/traces.json, kv_events.jsonl"
fi

echo ""
echo -e "\033[1;36m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

Output example (with changes):

━━━ BLIS Experiment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
⟫ 20260204-llama3-8b-chatsweep

  Model:     meta-llama/Llama-3-8B-Instruct
  Workload:  chatsweep (70→215 tokens, 50 req)
  Namespace: jchen

  Changes from defaults:
    TP: 1 → 2
    vLLM args: --trust-remote-code (added)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Output example (all defaults):

━━━ BLIS Experiment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
⟫ 20260204-llama3-8b-chatsweep

  Model:     meta-llama/Llama-3-8B-Instruct
  Workload:  chatsweep (70→215 tokens, 50 req)
  Namespace: jchen

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Then ask: "Deploy? [Y/n]" (single confirmation)

Phase 4: Deploy

Show progress with colored spinners/status.

IMPORTANT: All operations below only READ from tekton/ and tektonc/ directories. All generated files go to results/${EXPERIMENT_ID}/.

CRITICAL BUG FIX: The pipelinerun.yaml generation MUST update the params section to match user input. The base template contains hardcoded values that will override values.yaml if not updated. See Phase 3 setup for correct implementation.

# Cleanup existing resources if any
if kubectl get pipelinerun ${EXPERIMENT_ID} -n ${NAMESPACE} &>/dev/null; then
  echo -e "\033[34m⠋\033[0m Cleaning up existing pipelinerun..."
  kubectl delete pipelinerun ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false &>/dev/null
  sleep 2
fi
if kubectl get pipeline ${EXPERIMENT_ID} -n ${NAMESPACE} &>/dev/null; then
  kubectl delete pipeline ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false &>/dev/null
fi

# Apply RBAC: namespaced base (required) + cluster augment (best-effort).
# tektonc-data-collection#48 split the monolithic roles.yaml into:
#   - roles-ns.yaml: namespaced base (always required)
#   - roles-cluster.yaml: cluster augment (cluster-admin only)
# The cluster augment unlocks `kubectl get nodes`, cross-ns DCGM exec, and
# admin-mode prerequisites; missing it is non-fatal (degrades silently).
echo -e "\033[34m⠋\033[0m Applying RBAC..."
export NAMESPACE=${NAMESPACE}
envsubst < tekton/roles-ns.yaml | kubectl apply -f - >/dev/null 2>&1
envsubst < tekton/roles-cluster.yaml | kubectl apply -f - >/dev/null 2>&1 || true
if ! kubectl get serviceaccount helm-installer -n ${NAMESPACE} &>/dev/null; then
  echo -e "\033[1;31m✗\033[0m Failed to create helm-installer service account"
  echo -e "  \033[90m→ Check RBAC permissions and tekton/roles-ns.yaml\033[0m"
  exit 1
fi
echo -e "\033[32m✓\033[0m RBAC applied (helm-installer SA verified)"

# Apply Tekton tasks
echo -e "\033[34m⠋\033[0m Applying Tekton tasks..."
for step in tekton/steps/*.yaml; do kubectl apply -f "$step" >/dev/null 2>&1; done
for task in tekton/tasks/*.yaml; do kubectl apply -f "$task" >/dev/null 2>&1; done
echo -e "\033[32m✓\033[0m Tasks applied"

# Build pipeline
echo -e "\033[34m⠋\033[0m Building pipeline..."
source venv/bin/activate
python tektonc/tektonc.py \
  -t ${TEMPLATE_DIR}/data_pipeline.yaml.j2 \
  -f results/${EXPERIMENT_ID}/values.yaml \
  -r results/${EXPERIMENT_ID}/pipelinerun.yaml \
  -o results/${EXPERIMENT_ID}/pipeline.yaml 2>/dev/null
echo -e "\033[32m✓\033[0m Pipeline built"

# Deploy
echo -e "\033[34m⠋\033[0m Deploying..."
kubectl apply -f results/${EXPERIMENT_ID}/pipeline.yaml >/dev/null 2>&1
kubectl apply -f results/${EXPERIMENT_ID}/pipelinerun.yaml >/dev/null 2>&1
echo -e "\033[32m✓\033[0m \033[1;37mDeployed\033[0m"

# Check for GPU scheduling failures (wait up to 60s for pods to schedule)
echo -e "\033[34m⠋\033[0m Verifying GPU scheduling..."
sleep 10  # Initial wait for pods to be created

GPU_FAIL=false
for i in {1..5}; do
  # Check for Unschedulable pods with GPU resource issues
  PENDING_PODS=$(kubectl get pods -n ${NAMESPACE} -l tekton.dev/pipelineRun=${EXPERIMENT_ID} \
    --field-selector=status.phase=Pending -o jsonpath='{.items[*].metadata.name}' 2>/dev/null)

  if [ -n "${PENDING_PODS}" ]; then
    for pod in ${PENDING_PODS}; do
      EVENTS=$(kubectl get events -n ${NAMESPACE} --field-selector involvedObject.name=${pod} \
        -o jsonpath='{.items[*].message}' 2>/dev/null)
      if echo "${EVENTS}" | grep -qi "Insufficient nvidia.com/gpu\|gpu.*unavailable\|FailedScheduling.*gpu"; then
        GPU_FAIL=true
        break 2
      fi
    done
  fi

  # If no pending pods, scheduling succeeded
  if [ -z "${PENDING_PODS}" ]; then
    break
  fi
  sleep 10
done

if [ "${GPU_FAIL}" = true ]; then
  echo -e "\033[1;31m✗\033[0m GPU scheduling failed"

  # Terminate experiment
  kubectl delete pipelinerun ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false >/dev/null 2>&1
  kubectl delete pipeline ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false >/dev/null 2>&1

  echo -e "\033[1;31m━━━ GPU Unavailable ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
  echo -e "\033[1;31m✗\033[0m \033[1;37m${EXPERIMENT_ID}\033[0m terminated - GPUs no longer available"
  echo ""
  echo -e "  \033[34mRequired:\033[0m ${TP} GPU(s)"
  echo -e "  \033[34mStatus:\033[0m   GPUs were claimed by another workload"
  echo ""
  echo -e "  \033[33mOptions:\033[0m"
  echo -e "    \033[1;37m1.\033[0m Wait and retry: \033[90m/blis retry ${EXPERIMENT_ID}\033[0m"
  echo -e "    \033[1;37m2.\033[0m Check GPU availability: \033[90mkubectl describe nodes | grep -A5 'Allocated resources'\033[0m"
  echo -e "\033[1;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
  exit 1
fi

echo -e "\033[32m✓\033[0m GPU scheduling verified"

Output:

✓ RBAC applied
✓ Tasks applied
✓ Pipeline built
✓ Deployed
✓ GPU scheduling verified

Phase 5: Monitor (Background)

Launch background agent and show user how to check status:

echo ""
echo -e "\033[34mMonitoring:\033[0m Running in background"
echo -e "  \033[90mWatch:\033[0m   tkn pr logs \033[35m${EXPERIMENT_ID}\033[0m -n ${NAMESPACE} -f"
echo -e "  \033[90mStatus:\033[0m  tkn pr describe \033[35m${EXPERIMENT_ID}\033[0m -n ${NAMESPACE}"
echo -e "  \033[90mOutput:\033[0m  results/\033[35m${EXPERIMENT_ID}\033[0m/"

Launch background agent:

Task tool with:
- subagent_type: "general-purpose"
- run_in_background: true
- prompt: "Monitor PipelineRun ${EXPERIMENT_ID} in namespace ${NAMESPACE}.
  Poll every 30 seconds using 'tkn pr describe ${EXPERIMENT_ID} -n ${NAMESPACE}'.
  When status changes to Succeeded/Failed/Cancelled, save summary to results/${EXPERIMENT_ID}/monitoring.log
  and report back."

Phase 6: Download Results (On Success)

IMPORTANT: This phase runs automatically when the background monitoring agent reports success.

When the background agent reports that the pipeline succeeded, immediately execute this phase to download results from the cluster PVC to local storage.

# Download data from cluster PVC
echo ""
echo -e "\033[1;36m━━━ Downloading Results ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[34m⠋\033[0m Downloading data from cluster..."

mkdir -p results/${EXPERIMENT_ID}

# Create temporary pod to access data-pvc
kubectl run data-copy-${EXPERIMENT_ID} \
  --image=busybox \
  --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"data-copy-'${EXPERIMENT_ID}'","image":"busybox","command":["sleep","300"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}],"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"data-pvc"}}]}}' \
  -n ${NAMESPACE} >/dev/null 2>&1

# Wait for pod to be ready
kubectl wait --for=condition=Ready pod/data-copy-${EXPERIMENT_ID} -n ${NAMESPACE} --timeout=60s >/dev/null 2>&1

# Copy data from PVC to local
kubectl cp ${NAMESPACE}/data-copy-${EXPERIMENT_ID}:/data/${EXPERIMENT_ID}/ results/${EXPERIMENT_ID}/ 2>/dev/null

# Cleanup temporary pod
kubectl delete pod data-copy-${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false >/dev/null 2>&1

echo -e "\033[32m✓\033[0m Data downloaded to results/\033[35m${EXPERIMENT_ID}\033[0m/"

# Optional: Cleanup cluster resources
echo -e "\033[34m⠋\033[0m Cleaning up cluster resources..."
kubectl delete pipelinerun ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false >/dev/null 2>&1
kubectl delete pipeline ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false >/dev/null 2>&1
echo -e "\033[32m✓\033[0m Cluster resources cleaned up"

Display completion summary:

echo ""
echo -e "\033[32m━━━ Experiment Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[32m✓\033[0m \033[1;37m${EXPERIMENT_ID}\033[0m finished successfully"
echo ""
echo -e "  \033[34mLocal Data:\033[0m  results/\033[35m${EXPERIMENT_ID}\033[0m/"
echo -e "  \033[34mS3 Backup:\033[0m   s3://${BUCKET}/${NAMESPACE}/${EXPERIMENT_ID}/"
echo ""

# If observability mode, show trace files and analysis commands
if [ "${OBSERVABILITY_MODE}" = "true" ]; then
  echo -e "  \033[34mObservability Data:\033[0m"

  # List downloaded trace files
  if [ -f "results/${EXPERIMENT_ID}/traces.json" ]; then
    echo -e "    \033[32m✓\033[0m OTEL traces (traces.json)"
  fi
  if [ -f "results/${EXPERIMENT_ID}/kv_events.jsonl" ]; then
    echo -e "    \033[32m✓\033[0m KV events (kv_events.jsonl)"
  fi
  if [ -f "results/${EXPERIMENT_ID}/guidellm-results.json" ]; then
    echo -e "    \033[32m✓\033[0m Benchmark results (guidellm-results.json)"
  fi

  echo ""
  echo -e "  \033[90mQuick analysis:\033[0m"
  echo -e "  \033[90m  # Trace span types:\033[0m"
  echo -e "  \033[90m  jq '.resourceSpans[].scopeSpans[].spans[].name' results/${EXPERIMENT_ID}/traces.json | sort | uniq -c\033[0m"
  echo ""
  echo -e "  \033[90m  # KV event types:\033[0m"
  echo -e "  \033[90m  cat results/${EXPERIMENT_ID}/kv_events.jsonl | jq -r '.[1][][0]' | sort | uniq -c\033[0m"
fi

echo ""
echo -e "  \033[90mView results: /blis-results-viewer ${EXPERIMENT_ID}\033[0m"
echo -e "\033[32m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

---

Error Handling (Colored)

Pre-flight Failures

If critical checks fail, show actionable fixes:

echo -e "\033[1;31m━━━ Pre-flight Failed ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo ""
echo -e "  \033[1;31m✗\033[0m \033[34mhf-secret\033[0m missing"
echo -e "    \033[90m→ kubectl create secret generic hf-secret --from-literal=HF_TOKEN=hf_xxx -n ${NAMESPACE}\033[0m"
echo ""
echo -e "  \033[1;31m✗\033[0m \033[34mmodel-pvc\033[0m not found"
echo -e "    \033[90m→ See tekton/pvcs/model-pvc.yaml for template\033[0m"
echo ""
echo -e "\033[1;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

vLLM Deployment Failures

echo -e "\033[1;31m━━━ vLLM Error ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[31mCUDA out of memory\033[0m"
echo ""
echo -e "  \033[34mCurrent:\033[0m TP=${TP}, max_model_len=${MAX_MODEL_LEN}"
echo ""
echo -e "  \033[33mFixes:\033[0m"
echo -e "    \033[1;37m1.\033[0m Increase TP: ${TP} → $((TP*2)) \033[90m(doubles GPU memory)\033[0m"
echo -e "    \033[1;37m2.\033[0m Reduce context: ${MAX_MODEL_LEN} → $((MAX_MODEL_LEN/2))"
echo -e "    \033[1;37m3.\033[0m Reduce batch: max_num_seqs ${MAX_NUM_SEQS} → $((MAX_NUM_SEQS/2))"
echo ""
echo -e "\033[1;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

Then use AskUserQuestion:

questions:
  - question: "How do you want to fix the OOM error?"
    header: "Recovery"
    options:
      - label: "Increase TP (Recommended)"
        description: "TP=2 → TP=4, doubles GPU memory"
      - label: "Reduce context length"
        description: "max_model_len 8192 → 4096"
      - label: "Reduce batch size"
        description: "max_num_seqs 256 → 128"

Pipeline Task Failure

echo -e "\033[1;31m━━━ Task Failed ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[34mTask:\033[0m     \033[31m${FAILED_TASK}\033[0m"
echo -e "\033[34mReason:\033[0m   ${FAILURE_REASON}"
echo ""
echo -e "\033[90mLogs: tkn tr logs ${TASKRUN_NAME} -n ${NAMESPACE}\033[0m"
echo -e "\033[1;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

---

Completion States

NOTE: The completion logic below is implemented in Phase 6 of the workflow. This section serves as reference documentation.

Success

Download data from cluster PVC:

echo -e "\033[34m⠋\033[0m Downloading data from cluster..."
mkdir -p results/${EXPERIMENT_ID}

# Create temporary pod to access data-pvc
kubectl run data-copy-${EXPERIMENT_ID} \
  --image=busybox \
  --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"data-copy-'${EXPERIMENT_ID}'","image":"busybox","command":["sleep","300"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}],"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"data-pvc"}}]}}' \
  -n ${NAMESPACE} >/dev/null 2>&1
kubectl wait --for=condition=Ready pod/data-copy-${EXPERIMENT_ID} -n ${NAMESPACE} --timeout=60s >/dev/null 2>&1

# Copy data from PVC to local
kubectl cp ${NAMESPACE}/data-copy-${EXPERIMENT_ID}:/data/${EXPERIMENT_ID}/ results/${EXPERIMENT_ID}/ 2>/dev/null

# Cleanup temporary pod
kubectl delete pod data-copy-${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false >/dev/null 2>&1

echo -e "\033[32m✓\033[0m Data downloaded to results/\033[35m${EXPERIMENT_ID}\033[0m/"

Cleanup cluster resources:

echo -e "\033[34m⠋\033[0m Cleaning up cluster resources..."
kubectl delete pipelinerun ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false >/dev/null 2>&1
kubectl delete pipeline ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false >/dev/null 2>&1
echo -e "\033[32m✓\033[0m Cluster resources cleaned up"

Display completion:

echo -e "\033[32m━━━ Experiment Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[32m✓\033[0m \033[1;37m${EXPERIMENT_ID}\033[0m finished successfully"
echo ""
echo -e "  \033[34mData:\033[0m    results/\033[35m${EXPERIMENT_ID}\033[0m/"
echo -e "  \033[34mS3:\033[0m      s3://${BUCKET}/${NAMESPACE}/${EXPERIMENT_ID}/"

# If observability mode, show trace extraction commands
if [ "${OBSERVABILITY_MODE}" = "true" ]; then
  echo ""
  echo -e "  \033[34mTraces:\033[0m  results/\033[35m${EXPERIMENT_ID}\033[0m/traces.json"
  echo -e "  \033[34mKV events:\033[0m results/\033[35m${EXPERIMENT_ID}\033[0m/kv_events.jsonl"
  echo ""
  echo -e "  \033[90mAnalyze traces:\033[0m"
  echo -e "  \033[90m  jq '.resourceSpans[].scopeSpans[].spans[].name' results/${EXPERIMENT_ID}/traces.json | sort | uniq -c\033[0m"
  echo -e "  \033[90mCount KV events:\033[0m"
  echo -e "  \033[90m  cat results/${EXPERIMENT_ID}/kv_events.jsonl | jq -r '.[1][][0]' | sort | uniq -c\033[0m"
fi

echo ""
echo -e "  \033[90mCleanup: tkn pr delete ${EXPERIMENT_ID} -n ${NAMESPACE} -f\033[0m"
echo -e "\033[32m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

Failure

echo -e "\033[1;31m━━━ Experiment Failed ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[1;31m✗\033[0m \033[1;37m${EXPERIMENT_ID}\033[0m failed at \033[31m${FAILED_TASK}\033[0m"
echo ""
echo -e "  \033[90mLogs:  tkn pr logs ${EXPERIMENT_ID} -n ${NAMESPACE}\033[0m"
echo -e "  \033[90mRetry: /blis retry ${EXPERIMENT_ID}\033[0m"
echo -e "\033[1;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

---

Important Rules

1. NEVER modify anything in tekton/ directory (tasks, steps, roles)
2. NEVER modify anything in tektonc/ directory (compiler code)
3. NEVER modify original values.yaml or pipelinerun.yaml files in tektoncsample/
4. ALWAYS use results/${EXPERIMENT_ID}/ for generated files
5. ALWAYS use colored output with the defined color scheme
6. MINIMIZE user prompts - gather info in 1-2 consolidated questions
7. SHOW only changes from defaults, not full config
8. RUN pre-flight checks silently, summarize in one line
9. WARN about large models inline, not as separate step
10. USE background agent for monitoring

Quick Examples

# Full specification (no prompts needed)
/blis llama3-8b chatsweep in jchen with TP=2

# Minimal (will prompt for namespace only)
/blis llama3-8b chatsweep

# With custom vLLM args
/blis qwen-7b codesweep --trust-remote-code

# Custom workload
/blis mistral-7b custom prompt=500 output=100 in blis-dev

# Observability mode (journey + step + KV tracing)
/blis llama3-8b chatsweep --observability

# Observability with specific tracing
/blis mistral-7b codesweep --obs journey step

# Custom tracing sample rate
/blis qwen-7b custom --observability step-rate=0.05