deploy-otel

# Deploy OTEL Observability Stack Deploy a complete OpenTelemetry observability stack to a Kind cluster for testing ToolHives telemetry capabilities. ## Steps ### 1. Verify Prerequisites Check that required tools are installed: ```bash echo "Checking prerequisites..." command -v kind >/dev/null 2>&1 || { echo "ERROR: kind is not installed"; exit 1; } command -v helm >/dev/null 2>&1 || { echo "ERROR: helm is not installed"; exit 1; } command -v kubectl >/dev/null 2>&1 || { echo "ERROR: kubectl is not installed"; exit 1; } echo "All prerequisites met." ``` ### 2. Create Kind Cluster Create the Kind cluster if it doesn't exist: ```bash CLUSTER_NAME="toolhive" if kind get clusters 2>/dev/null | grep -q "^${CLUSTER_NAME}$"; then echo "Kind cluster '${CLUSTER_NAME}' already exists" else echo "Creating Kind cluster '${CLUSTER_NAME}'..." kind create cluster --name ${CLUSTER_NAME} fi # Export kubeconfig kind get kubeconfig --name ${CLUSTER_NAME} > kconfig.yaml echo "Kubeconfig written to kconfig.yaml" ``` ### 3. Add Helm Repositories ```bash echo "Adding Helm repositories..." helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo add grafana https://grafana.github.io/helm-charts helm repo update echo "Helm repositories updated." ``` ### 4. Install Prometheus/Grafana Stack ```bash echo "Installing kube-prometheus-stack..." helm upgrade -i kube-prometheus-stack prometheus-community/kube-prometheus-stack \ -f examples/otel/prometheus-stack-values.yaml \ -n monitoring --create-namespace \ --kubeconfig kconfig.yaml \ --wait --timeout 5m echo "Prometheus/Grafana stack installed." ``` ### 5. Install Tempo for Distributed Tracing ```bash echo "Installing Grafana Tempo..." helm upgrade -i tempo grafana/tempo \ -f examples/otel/tempo-values.yaml \ -n monitoring \ --kubeconfig kconfig.yaml \ --wait --timeout 3m echo "Grafana Tempo installed." ``` ### 6. Install OpenTelemetry Collector ```bash echo "Installing OpenTelemetry Collector..." helm upgrade -i otel-collector open-telemetry/opentelemetry-collector \ -f examples/otel/otel-values.yaml \ -n monitoring \ --kubeconfig kconfig.yaml \ --wait --timeout 3m echo "OpenTelemetry Collector installed." ``` ### 7. Verify Deployment ```bash echo "Verifying deployment..." kubectl get pods -n monitoring --kubeconfig kconfig.yaml ``` ### 8. Display Access Instructions ```bash cat <<'EOF' === OTEL Stack Deployment Complete === To access the UIs, run these port-forward commands: # Grafana (admin / admin) kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:3000 --kubeconfig kconfig.yaml # Prometheus kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 --kubeconfig kconfig.yaml EOF ``` ## Troubleshooting If Helm installations fail due to incompatible values, it may be because the Helm charts have been updated and our `values.yaml` files are no longer compatible. **Chart Documentation:** - OpenTelemetry Collector: https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector - Prometheus Stack: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack - Tempo: https://github.com/grafana/helm-charts/tree/main/charts/tempo **If you encounter issues:** 1. Check the chart's `values.yaml` for schema changes in the versions of the Charts we are using 2. Compare with our values files in `examples/otel/` 3. Create an issue at: https://github.com/stacklok/toolhive/issues describing what the issue is and recommend a fix ## What This Deploys | Component | Description | |-----------|-------------| | Prometheus | Metrics storage, scrapes OTEL collector on port 8889 | | Grafana | Visualization dashboards (admin/admin) | | Tempo | Distributed tracing backend, receives traces from OTEL Collector | | OTEL Collector | Receives OTLP metrics/traces, exports to Prometheus and Tempo | ## Cleanup To remove everything: ```bash task kind-destroy ``` Or manually: ```bash kind delete cluster --name toolhive rm -f kconfig.yaml ```