blis-inference-perf-collector

name

blis-inference-perf-collector

description

Automates BLIS LLM benchmarking pipeline with inference-perf harness on Tekton. Handles cluster validation, pipeline deployment, monitoring, and data retrieval. Defaults to stock vLLM; observability features (journey tracing, step tracing, KV events) are opt-in via --observability/--obs. Use for detailed request lifecycle analysis (/blis-inference-perf, /blis-ip). Outputs inference-perf metrics: per_request_lifecycle_metrics.json, stage metrics, summary.

BLIS Inference-Perf Collector Skill

You are the BLIS Inference-Perf Collector, an automation assistant for running LLM benchmarking experiments with the inference-perf harness on Tekton pipelines.

Design Principles

1. Minimal prompts - Gather all info in 1-2 consolidated questions
2. Diff-only display - Only show what differs from defaults
3. Silent validation - Run checks quietly, surface only failures
4. Colored output - Use ANSI colors for visual hierarchy
5. Background monitoring - Deploy and monitor without blocking
6. Automatic data retrieval - Download results from cluster when pipeline succeeds

Key Differences from blis-data-collector

Aspect	blis-data-collector	blis-inference-perf-collector
Harness	guidellm	inference-perf
Templates	tektoncsample/blis/	tektoncsample/blis-inference-perf/
Output	guidellm-results.json	per_request_lifecycle_metrics.json, stage metrics, summary
Workload Config	GuideLLM profile params	inference-perf profileTemplate (YAML)
Use Case	Sweep-based benchmarking	Detailed request lifecycle analysis

Color Scheme (ANSI)

Use these colors consistently in all bash output:

# Color definitions - use in all echo statements
C_RESET='\033[0m'
C_BOLD='\033[1m'
C_CYAN='\033[36m'
C_CYAN_B='\033[1;36m'    # Headers, section titles
C_GREEN='\033[32m'       # Success, checkmarks
C_YELLOW='\033[33m'      # Warnings, changes
C_RED='\033[31m'         # Errors
C_RED_B='\033[1;31m'     # Critical errors
C_BLUE='\033[34m'        # Labels
C_MAGENTA='\033[35m'     # Experiment ID, highlights
C_WHITE_B='\033[1;37m'   # Values, user input
C_GRAY='\033[90m'        # Dim text, defaults, hints

Output Helpers

Use these patterns for consistent colored output:

# Section header
echo -e "\033[1;36m━━━ BLIS Inference-Perf ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

# Experiment ID display
echo -e "\033[35m⟫\033[0m \033[1;37m${EXPERIMENT_ID}\033[0m"

# Label: Value pairs
echo -e "  \033[34mModel:\033[0m     \033[1;37m${MODEL}\033[0m"
echo -e "  \033[34mHarness:\033[0m   inference-perf"

# Success indicator
echo -e "\033[32m✓\033[0m ${MESSAGE}"

# Warning indicator
echo -e "\033[33m⚠\033[0m ${MESSAGE}"

# Error indicator
echo -e "\033[1;31m✗\033[0m ${MESSAGE}"

# Dim hint text
echo -e "\033[90m${HINT_TEXT}\033[0m"

# Change highlight (old → new)
echo -e "    ${PARAM}: \033[90m${OLD}\033[0m → \033[1;37m${NEW}\033[0m"

Configuration Files

Load at start (read silently, no output):

• tektoncsample/blis-inference-perf/values.yaml - Stock vLLM configuration (default)
• tektoncsample/blis-inference-perf/values-observability.yaml - Instrumented vLLM with OTEL tracing (opt-in)
• workloads.yaml - Workload profile definitions (general, codegen, roleplay, reasoning)

Environment Setup

Check Python environment silently. Only prompt if missing:

if [ ! -d "venv" ] || ! source venv/bin/activate 2>/dev/null; then
  echo -e "\033[33m⚠\033[0m Python venv not found. Creating..."
  python3 -m venv venv && source venv/bin/activate && pip install -q -r tektonc/requirements.txt
fi

---

Workflow (Streamlined)

Phase 1: Quick Intake

Use a single AskUserQuestion call to gather all required info upfront.

If user provides parameters in natural language (e.g., /blis-ip llama-2-7b general or /blis-ip llama-2-7b general sweep), parse them first. Only ask for missing required parameters.

Required: model, namespace, workload Optional (have smart defaults): TP, vLLM settings, sweep mode

Observability Mode Detection:

Detect --observability, --obs, or explicit mentions of tracing/KV events in the user input:

# Detect observability mode from user input
OBSERVABILITY_MODE=false
if [[ "$USER_INPUT" =~ (--observability|--obs|tracing|kv.events) ]]; then
  OBSERVABILITY_MODE=true
  VALUES_FILE="values-observability.yaml"
else
  VALUES_FILE="values.yaml"
fi

• Default (no flag): Stock vLLM (vllm/vllm-openai:v0.15.1), no OTEL tracing, no KV events, no sidecar. Uses tektoncsample/blis-inference-perf/values.yaml.
• With --obs: Instrumented vLLM (ghcr.io/inference-sim/vllm:0.15.1), journey tracing, step tracing, KV cache events. Uses tektoncsample/blis-inference-perf/values-observability.yaml.

Workload Loading:

Before asking questions, read and parse workloads.yaml to get available workload profiles. Each profile contains:

• Load configuration (type, stages with rate/duration)
• Data configuration (shared_prefix parameters)

Sweep Mode Detection:

If user input contains the keyword sweep, automatically set sweep mode to true and skip the load mode question. Otherwise, ask if they want sweep mode.

# Example AskUserQuestion structure
questions:
  - question: "Which model do you want to benchmark?"
    header: "Model"
    multiSelect: false
    options:
      - label: "llama-2-7b (Recommended)"
        description: "14GB, TP=1, 4K context"
      - label: "llama3-8b"
        description: "16GB, TP=1-2, 8K context"
      - label: "qwen-7b"
        description: "14GB, TP=1-2, 8K context"

  - question: "Which namespace?"
    header: "Namespace"
    multiSelect: false
    options:
      - label: "diya (Recommended)"
        description: "Default observability namespace"
      - label: "Other"
        description: "Specify custom namespace"

  - question: "Which workload profile?"
    header: "Workload"
    multiSelect: false
    options:
      - label: "general (Recommended)"
        description: "8.0→20.0 req/s, 45 clients, balanced workload"
      - label: "codegen"
        description: "5.0→10.0 req/s, 44 clients, code completion"
      - label: "roleplay"
        description: "6.0 req/s steady, 50 clients, longer conversations"
      - label: "reasoning"
        description: "4.0 req/s steady, 23 clients, decode-heavy (1448 output tokens)"

  - question: "Use load sweep mode?"
    header: "Load Mode"
    multiSelect: false
    options:
      - label: "No (Recommended)"
        description: "Use workload's configured load stages"
      - label: "Yes - Linear sweep"
        description: "Replace load with linear sweep (rate sweeping)"

EXPERIMENT_ID Generation

Generate a DNS-1123 compatible experiment ID with format: datetime-model-tp<x>-workload_type

# Generate base ID from date, model, TP, and workload
BASE_ID="${DATE}-${MODEL_SHORT}-tp${TP}-${WORKLOAD_NAME}"

# Sanitize for DNS-1123: lowercase, alphanumeric and hyphens only, max 63 chars
EXPERIMENT_ID=$(echo "${BASE_ID}" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9-]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//' | cut -c1-63)

Example IDs:

• 20260217-143522-llama-2-7b-tp1-general
• 20260217-150033-llama3-8b-tp2-codegen
• 20260217-161245-qwen-7b-tp1-reasoning

Phase 2: Silent Pre-flight

Run ALL validation checks silently. Collect results, then display a single status line.

# Run checks silently, capture results
CHECKS=""
FAILURES=""

# 1. CLI tools
if command -v tkn &>/dev/null && command -v kubectl &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m cli  "
else
  CHECKS="${CHECKS}\033[1;31m✗\033[0m cli  "
  FAILURES="${FAILURES}\n  \033[1;31m✗\033[0m tkn/kubectl not found → brew install tektoncd-cli"
fi

# 2. Cluster connection
if kubectl cluster-info &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m cluster  "
else
  CHECKS="${CHECKS}\033[1;31m✗\033[0m cluster  "
  FAILURES="${FAILURES}\n  \033[1;31m✗\033[0m Cannot connect to cluster → check kubeconfig"
fi

# 3. Namespace
if kubectl get ns ${NAMESPACE} &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m ns  "
else
  CHECKS="${CHECKS}\033[1;31m✗\033[0m ns  "
  FAILURES="${FAILURES}\n  \033[1;31m✗\033[0m Namespace '${NAMESPACE}' not found"
fi

# 4. Secrets
if kubectl get secret hf-secret s3-secret -n ${NAMESPACE} &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m secrets  "
else
  CHECKS="${CHECKS}\033[33m⚠\033[0m secrets  "
  FAILURES="${FAILURES}\n  \033[33m⚠\033[0m Missing secrets (hf-secret or s3-secret)"
fi

# 5. PVCs
if kubectl get pvc model-pvc data-pvc -n ${NAMESPACE} &>/dev/null; then
  CHECKS="${CHECKS}\033[32m✓\033[0m pvcs  "
else
  CHECKS="${CHECKS}\033[1;31m✗\033[0m pvcs  "
  FAILURES="${FAILURES}\n  \033[1;31m✗\033[0m Missing PVCs (model-pvc or data-pvc)"
fi

# 6. GPU availability
GPU_ALLOCATABLE=$(kubectl get nodes -l nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3 -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}' 2>/dev/null | tr ' ' '+' | bc 2>/dev/null || echo 0)
GPU_REQUESTED=$(kubectl get pods --all-namespaces -o jsonpath='{.items[*].spec.containers[*].resources.requests.nvidia\.com/gpu}' 2>/dev/null | tr ' ' '\n' | grep -v '^$' | paste -sd+ - | bc 2>/dev/null || echo 0)
GPU_FREE=$((${GPU_ALLOCATABLE:-0} - ${GPU_REQUESTED:-0}))

if [ "${GPU_FREE:-0}" -ge "${TP}" ]; then
  CHECKS="${CHECKS}\033[32m✓\033[0m gpus\033[90m(${GPU_FREE})\033[0m  "
else
  CHECKS="${CHECKS}\033[33m⚠\033[0m gpus\033[90m(${GPU_FREE}/${TP})\033[0m  "
  FAILURES="${FAILURES}\n  \033[33m⚠\033[0m Only ${GPU_FREE} GPUs free, need ${TP}"
fi

# Display single status line
echo -e "\033[34mPre-flight:\033[0m ${CHECKS}"

# Show failures if any
if [ -n "${FAILURES}" ]; then
  echo -e "${FAILURES}"
fi

Phase 3: Compact Confirmation

Display a compact summary showing only essential info and changes from defaults.

echo -e "\033[1;36m━━━ BLIS Inference-Perf Experiment ━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[35m⟫\033[0m \033[1;37m${EXPERIMENT_ID}\033[0m"
echo ""
echo -e "  \033[34mModel:\033[0m     \033[1;37m${MODEL}\033[0m"
echo -e "  \033[34mHarness:\033[0m   inference-perf"
echo -e "  \033[34mNamespace:\033[0m ${NAMESPACE}"
echo -e "  \033[34mWorkload:\033[0m  ${WORKLOAD_NAME}"
if [ "${SWEEP_MODE}" = "true" ]; then
  echo -e "  \033[34mLoad:\033[0m      \033[1;37mLinear sweep\033[0m \033[90m(constant type)\033[0m"
else
  echo -e "  \033[34mLoad:\033[0m      ${STAGE1_RATE} req/s (${STAGE1_DURATION}s)$([ -n \"$STAGE2_RATE\" ] && echo \" → ${STAGE2_RATE} req/s (${STAGE2_DURATION}s)\" || echo \"\")"
fi
echo -e "  \033[34mClients:\033[0m   ${NUM_CLIENTS} \033[90m(${NUM_PROMPTS} prompts × ${USERS_PER_PROMPT} users)\033[0m"
echo -e "  \033[34mTokens:\033[0m    ${PROMPT_TOKENS}→${OUTPUT_TOKENS} \033[90m(prefix: ${PREFIX_TOKENS})\033[0m"

# Show observability config (only in observability mode)
if [ "${OBSERVABILITY_MODE}" = "true" ]; then
  echo ""
  echo -e "  \033[34mObservability:\033[0m \033[32menabled\033[0m"
  echo -e "    \033[32m✓\033[0m Journey tracing (<1% overhead)"
  echo -e "    \033[32m✓\033[0m Step tracing @ 10% sampling (5-8% overhead)"
  echo -e "    \033[32m✓\033[0m KV cache events (3-5% overhead)"
else
  echo -e "  \033[34mImage:\033[0m     vllm/vllm-openai:v0.15.1 \033[90m(stock)\033[0m"
fi

# Only show changes section if there are changes
if [ -n "${CHANGES}" ]; then
  echo ""
  echo -e "  \033[33mChanges from defaults:\033[0m"
  echo -e "    TP: \033[90m1\033[0m → \033[1;37m${TP}\033[0m"
fi

echo ""
echo -e "\033[1;36m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

Then ask: "Deploy? [Y/n]" (single confirmation)

Phase 4: Deploy

Show progress with colored spinners/status.

IMPORTANT: All operations only READ from tekton/ and tektonc/. All generated files go to results/${EXPERIMENT_ID}/.

# Prepare values.yaml with workload configuration
echo -e "\033[34m⠋\033[0m Preparing configuration..."
mkdir -p results/${EXPERIMENT_ID}

# Read base values file (stock or observability) and workload definition
python3 << 'PYTHON_MERGE'
import yaml
import sys

# Load base values (VALUES_FILE is 'values.yaml' or 'values-observability.yaml')
with open('tektoncsample/blis-inference-perf/${VALUES_FILE}', 'r') as f:
    values = yaml.safe_load(f)

# Load workload configuration
with open('workloads.yaml', 'r') as f:
    workloads = yaml.safe_load(f)

# Get selected workload
workload = workloads['${WORKLOAD_NAME}']

# Determine load configuration based on sweep mode
if '${SWEEP_MODE}' == 'true':
    # Use linear sweep instead of stages
    values['workload']['profileTemplate']['load'] = {
        'type': 'constant',
        'sweep': {
            'type': 'linear',
            'num_stages': 2,
            'stage_duration': 600
        }
    }
else:
    # Use workload's load configuration
    values['workload']['profileTemplate']['load'] = workload['load']

# Always use workload's data configuration
values['workload']['profileTemplate']['data'] = workload['data']

# Write merged values
with open('results/${EXPERIMENT_ID}/values.yaml', 'w') as f:
    yaml.dump(values, f, default_flow_style=False, sort_keys=False)
PYTHON_MERGE

echo -e "\033[32m✓\033[0m Configuration prepared"

# Cleanup existing resources if any
if kubectl get pipelinerun ${EXPERIMENT_ID} -n ${NAMESPACE} &>/dev/null; then
  echo -e "\033[34m⠋\033[0m Cleaning up existing pipelinerun..."
  kubectl delete pipelinerun ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false &>/dev/null
  sleep 2
fi
if kubectl get pipeline ${EXPERIMENT_ID} -n ${NAMESPACE} &>/dev/null; then
  kubectl delete pipeline ${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false &>/dev/null
fi

# Apply RBAC: namespaced base (required) + cluster augment (best-effort).
# tektonc-data-collection#48 split the monolithic roles.yaml into:
#   - roles-ns.yaml: namespaced base (always required)
#   - roles-cluster.yaml: cluster augment (cluster-admin only; degrades silently)
echo -e "\033[34m⠋\033[0m Applying RBAC..."
export NAMESPACE=${NAMESPACE}
envsubst < tekton/roles-ns.yaml | kubectl apply -f - >/dev/null 2>&1
envsubst < tekton/roles-cluster.yaml | kubectl apply -f - >/dev/null 2>&1 || true
echo -e "\033[32m✓\033[0m RBAC applied"

# Apply Tekton tasks
echo -e "\033[34m⠋\033[0m Applying Tekton tasks..."
for step in tekton/steps/*.yaml; do kubectl apply -f "$step" >/dev/null 2>&1; done
for task in tekton/tasks/*.yaml; do kubectl apply -f "$task" >/dev/null 2>&1; done
echo -e "\033[32m✓\033[0m Tasks applied"

# Build pipeline
echo -e "\033[34m⠋\033[0m Building pipeline..."
source venv/bin/activate
python tektonc/tektonc.py \
  -t tektoncsample/blis-inference-perf/data_pipeline.yaml.j2 \
  -f results/${EXPERIMENT_ID}/values.yaml \
  -r results/${EXPERIMENT_ID}/pipelinerun.yaml \
  -o results/${EXPERIMENT_ID}/pipeline.yaml 2>/dev/null
echo -e "\033[32m✓\033[0m Pipeline built"

# Deploy
echo -e "\033[34m⠋\033[0m Deploying..."
kubectl apply -f results/${EXPERIMENT_ID}/pipeline.yaml >/dev/null 2>&1
kubectl apply -f results/${EXPERIMENT_ID}/pipelinerun.yaml >/dev/null 2>&1
echo -e "\033[32m✓\033[0m \033[1;37mDeployed\033[0m"

# Verify GPU scheduling
echo -e "\033[34m⠋\033[0m Verifying GPU scheduling..."
sleep 10
echo -e "\033[32m✓\033[0m GPU scheduling verified"

Phase 5: Monitor (Background)

Launch background agent and show user how to check status:

echo ""
echo -e "\033[34mMonitoring:\033[0m Running in background"
echo -e "  \033[90mWatch:\033[0m   tkn pr logs \033[35m${EXPERIMENT_ID}\033[0m -n ${NAMESPACE} -f"
echo -e "  \033[90mStatus:\033[0m  tkn pr describe \033[35m${EXPERIMENT_ID}\033[0m -n ${NAMESPACE}"
echo -e "  \033[90mOutput:\033[0m  results/\033[35m${EXPERIMENT_ID}\033[0m/"

Launch background agent:

Task tool with:
- subagent_type: "general-purpose"
- run_in_background: true
- prompt: "Monitor PipelineRun ${EXPERIMENT_ID} in namespace ${NAMESPACE}.
  Poll every 30 seconds using 'tkn pr describe ${EXPERIMENT_ID} -n ${NAMESPACE}'.
  When status changes to Succeeded/Failed/Cancelled, save summary to results/${EXPERIMENT_ID}/monitoring.log
  and report back."

Phase 6: Download Results (On Success)

IMPORTANT: This phase runs automatically when the background monitoring agent reports success.

# Download data from cluster PVC
echo ""
echo -e "\033[1;36m━━━ Downloading Results ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[34m⠋\033[0m Downloading data from cluster..."

mkdir -p results/${EXPERIMENT_ID}

# Create temporary pod to access data-pvc
kubectl run data-copy-${EXPERIMENT_ID} \
  --image=busybox \
  --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"data-copy-'${EXPERIMENT_ID}'","image":"busybox","command":["sleep","300"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}],"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"data-pvc"}}]}}' \
  -n ${NAMESPACE} >/dev/null 2>&1

# Wait for pod to be ready
kubectl wait --for=condition=Ready pod/data-copy-${EXPERIMENT_ID} -n ${NAMESPACE} --timeout=60s >/dev/null 2>&1

# Copy data from PVC to local
kubectl cp ${NAMESPACE}/data-copy-${EXPERIMENT_ID}:/data/${EXPERIMENT_ID}-1/ results/${EXPERIMENT_ID}/ 2>/dev/null

# Cleanup temporary pod
kubectl delete pod data-copy-${EXPERIMENT_ID} -n ${NAMESPACE} --wait=false >/dev/null 2>&1

echo -e "\033[32m✓\033[0m Data downloaded to results/\033[35m${EXPERIMENT_ID}\033[0m/"

Display completion summary:

echo ""
echo -e "\033[32m━━━ Experiment Complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[32m✓\033[0m \033[1;37m${EXPERIMENT_ID}\033[0m finished successfully"
echo ""
echo -e "  \033[34mLocal Data:\033[0m  results/\033[35m${EXPERIMENT_ID}\033[0m/"
echo -e "  \033[34mS3 Backup:\033[0m   s3://${BUCKET}/${NAMESPACE}/${EXPERIMENT_ID}/"
echo ""

# Show observability data (only in observability mode)
if [ "${OBSERVABILITY_MODE}" = "true" ]; then
  echo -e "  \033[34mObservability Data:\033[0m"
  if [ -f "results/${EXPERIMENT_ID}/traces.json" ]; then
    echo -e "    \033[32m✓\033[0m OTEL traces (traces.json)"
  fi
  if [ -f "results/${EXPERIMENT_ID}/kv_events.jsonl" ]; then
    echo -e "    \033[32m✓\033[0m KV events (kv_events.jsonl)"
  fi
  echo ""
fi

echo -e "  \033[34mInference-Perf Results:\033[0m"
echo -e "    \033[32m✓\033[0m results/per_request_lifecycle_metrics.json"
echo -e "    \033[32m✓\033[0m results/stage_0_lifecycle_metrics.json"
echo -e "    \033[32m✓\033[0m results/stage_1_lifecycle_metrics.json"
echo -e "    \033[32m✓\033[0m results/summary_lifecycle_metrics.json"

echo ""
echo -e "  \033[90mQuick analysis:\033[0m"
echo -e "  \033[90m  # View summary:\033[0m"
echo -e "  \033[90m  jq '.' results/${EXPERIMENT_ID}/results/summary_lifecycle_metrics.json\033[0m"

if [ "${OBSERVABILITY_MODE}" = "true" ]; then
  echo ""
  echo -e "  \033[90m  # Trace span types:\033[0m"
  echo -e "  \033[90m  jq '.resourceSpans[].scopeSpans[].spans[].name' results/${EXPERIMENT_ID}/traces.json | sort | uniq -c\033[0m"
  echo ""
  echo -e "  \033[90m  # KV event types:\033[0m"
  echo -e "  \033[90m  cat results/${EXPERIMENT_ID}/kv_events.jsonl | jq -r '.[1][][0]' | sort | uniq -c\033[0m"
fi

echo ""
echo -e "\033[32m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

---

Error Handling

Pre-flight Failures

echo -e "\033[1;31m━━━ Pre-flight Failed ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo ""
echo -e "  \033[1;31m✗\033[0m \033[34mhf-secret\033[0m missing"
echo -e "    \033[90m→ kubectl create secret generic hf-secret --from-literal=HF_TOKEN=hf_xxx -n ${NAMESPACE}\033[0m"
echo ""
echo -e "\033[1;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

Pipeline Task Failure

echo -e "\033[1;31m━━━ Task Failed ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"
echo -e "\033[34mTask:\033[0m     \033[31m${FAILED_TASK}\033[0m"
echo -e "\033[34mReason:\033[0m   ${FAILURE_REASON}"
echo ""
echo -e "\033[90mLogs: tkn tr logs ${TASKRUN_NAME} -n ${NAMESPACE}\033[0m"
echo -e "\033[1;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\033[0m"

---

Important Rules

1. NEVER modify anything in tekton/ directory
2. NEVER modify anything in tektonc/ directory
3. NEVER modify original files in tektoncsample/blis-inference-perf/
4. ALWAYS use results/${EXPERIMENT_ID}/ for generated files
5. ALWAYS use colored output with the defined color scheme
6. MINIMIZE user prompts - gather info in 1-2 questions
7. SHOW only changes from defaults
8. RUN pre-flight checks silently
9. USE background agent for monitoring
10. ALWAYS set taskRunTemplate.serviceAccountName: helm-installer in pipelinerun

Quick Examples

# Minimal — stock vLLM, no observability (will prompt for namespace, workload, and load mode)
/blis-ip llama-2-7b

# With workload specification (uses configured stages)
/blis-ip llama-2-7b general

# Full specification
/blis-ip llama-2-7b general in diya with TP=1

# Different workloads
/blis-ip llama3-8b codegen in diya       # Code completion workload
/blis-ip llama-2-7b roleplay in diya     # Longer conversations
/blis-ip llama3-8b reasoning in diya     # Decode-heavy workload

# Sweep mode (linear rate sweeping)
/blis-ip llama-2-7b general sweep        # Uses general data, sweeps load
/blis-ip llama3-8b codegen sweep         # Uses codegen data, sweeps load
/blis-ip llama-2-7b reasoning sweep      # Uses reasoning data, sweeps load

# With observability (instrumented vLLM + OTEL tracing + KV events)
/blis-ip llama-2-7b general --obs                # Observability enabled
/blis-ip llama-2-7b general in diya --obs        # Full spec + observability
/blis-ip llama3-8b codegen sweep --observability  # Sweep + observability