pwaf-connector-testing

<!-- skill-version: 1.0.0 --> # PWAF Connector Testing Use this skill when building or using a test runner for PWAF connector projects. The pattern here can be applied to any new connector project to test all connectors across all auth types. --- ## Overview A PWAF connector test runner (`tests/run_all_tests.sh`) validates every connector (REST API, Python SDK, SQL drivers, Go, Java, Node.js) across all supported auth types (PAT, OAuth M2M, OAuth U2M), with proper environment isolation for each test. Key design principles: 1. **Each test runs in a clean `env -i` subprocess** — never inherit full shell environment 2. **Each auth type has its own set of required variables** — never mix them 3. **Browser tests (U2M) are opt-in** — require `PWAF_RUN_BROWSER_TESTS=1` 4. **Build failures skip tests** — don't fail the run, just mark as skipped 5. **Results written to a report file** for review --- ## Environment File The test runner reads credentials from a `.env` file (passed with `--env <name>`). The file is located at `.env_<name>` in the project root. ```bash # .env_demo — example DATABRICKS_HOST=https://myworkspace.cloud.databricks.com DATABRICKS_TOKEN=dapiXXXXXXXXXXXXXXXX DATABRICKS_CLIENT_ID=11111111-2222-3333-4444-555555555555 DATABRICKS_CLIENT_SECRET=doceXXXXXXXXXXXXXXXX DATABRICKS_U2M_CLIENT_ID=66666666-7777-8888-9999-000000000000 DATABRICKS_U2M_CLIENT_SECRET=doceYYYYYYYYYYYYYYYY DATABRICKS_REDIRECT_URI=http://localhost:8080/callback DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/abcdef123456 DATABRICKS_WAREHOUSE_ID=abcdef123456 CATALOG=my_catalog ``` --- ## Auth Type → Required Variables Each auth type is tested in its own `env -i` subprocess with only these variables: | Auth type | Required env vars | |-----------|------------------| | `pat` | `DATABRICKS_HOST`, `DATABRICKS_TOKEN` | | `oauth_m2m` | `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET` | | `u2m_custom_oauth_app` | `DATABRICKS_HOST`, `DATABRICKS_U2M_CLIENT_ID`, `DATABRICKS_U2M_CLIENT_SECRET` (if confidential), `DATABRICKS_REDIRECT_URI` | | `u2m_token_env` | `DATABRICKS_HOST`, `DATABRICKS_ACCESS_TOKEN` (or `DATABRICKS_TOKEN`) | Always include in the `env -i` base: `PATH`, `HOME`, `USER`, `LANG`. > **Never include `DATABRICKS_AUTH_TYPE` in any test subprocess.** The Python SDK, Go SDK, and Java SDK all read it internally. Setting it to application-level values like `"oauth_m2m"` breaks credential resolution. > Use `APP_AUTH_TYPE=<value>` as the application-level selector passed to the connector. --- ## Script Structure ```bash #!/usr/bin/env bash set -euo pipefail # === Defaults === ENV_NAME="demo" FILTER_APP="" FILTER_AUTH="" PARALLEL=0 PARALLEL_AUTHS="pat,oauth_m2m" SKIP_BUILD=0 BROWSER_TESTS="${PWAF_RUN_BROWSER_TESTS:-0}" # === Argument parsing === while [[ $# -gt 0 ]]; do case "$1" in --env) ENV_NAME="$2"; shift 2 ;; --app) FILTER_APP="$2"; shift 2 ;; --auth) FILTER_AUTH="$2"; shift 2 ;; --parallel) PARALLEL=1; shift 1 ;; --parallel-auths) PARALLEL_AUTHS="$2"; shift 2 ;; --skip-build) SKIP_BUILD=1; shift 1 ;; *) echo "Unknown option: $1"; exit 1 ;; esac done ENV_FILE=".env_${ENV_NAME}" source "$ENV_FILE" ``` --- ## Build Phase Build each connector before running tests. Record build failures — tests for failed connectors are automatically skipped. ```bash build_connector() { local name="$1" local dir="connectors/$name" case "$name" in go-sdk|go-sql-driver) (cd "$dir" && go build ./...) ;; java-sdk|java-jdbc) (cd "$dir" && mvn -q package -DskipTests) ;; nodejs-sql-driver) (cd "$dir" && npm install && npm run build) ;; python-*|rest-api) true ;; # No build step for Python esac } ``` --- ## Test Execution with env -i Each connector test runs in a clean subprocess. The `run_connector_auth` function: 1. Checks if this connector supports this auth type 2. Skips browser tests unless `PWAF_RUN_BROWSER_TESTS=1` 3. Builds the `env -i` environment with only auth-specific vars 4. Runs the connector with a timeout 5. Records PASS/FAIL/SKIP ```bash run_connector_auth() { local connector="$1" local auth_type="$2" # Skip if build failed if build_failed "$connector"; then record_result "$connector" "$auth_type" "SKIP" 0 "build-failed" return fi # Skip browser tests unless enabled if is_browser_auth "$auth_type" && [[ "$BROWSER_TESTS" != "1" ]]; then record_result "$connector" "$auth_type" "SKIP" 0 "browser-disabled" return fi # Build env -i vars for this auth type local env_vars=( "PATH=$PATH" "HOME=$HOME" "USER=${USER:-}" "LANG=${LANG:-en_US.UTF-8}" "DATABRICKS_HOST=$DATABRICKS_HOST" "APP_AUTH_TYPE=$auth_type" ) case "$auth_type" in pat) env_vars+=("DATABRICKS_TOKEN=$DATABRICKS_TOKEN") ;; oauth_m2m) env_vars+=( "DATABRICKS_CLIENT_ID=$DATABRICKS_CLIENT_ID" "DATABRICKS_CLIENT_SECRET=$DATABRICKS_CLIENT_SECRET" ) ;; u2m_custom_oauth_app) env_vars+=( "DATABRICKS_U2M_CLIENT_ID=$DATABRICKS_U2M_CLIENT_ID" "DATABRICKS_U2M_CLIENT_SECRET=${DATABRICKS_U2M_CLIENT_SECRET:-}" "DATABRICKS_REDIRECT_URI=${DATABRICKS_REDIRECT_URI:-http://localhost:8080/callback}" ) ;; u2m_token_env) env_vars+=("DATABRICKS_ACCESS_TOKEN=$DATABRICKS_ACCESS_TOKEN") ;; esac # Add SQL-specific vars if connector needs them if connector_needs_sql "$connector"; then env_vars+=( "DATABRICKS_HTTP_PATH=$DATABRICKS_HTTP_PATH" "DATABRICKS_WAREHOUSE_ID=${DATABRICKS_WAREHOUSE_ID:-}" ) fi # Add Java-specific vars if [[ "$connector" == java-* ]]; then env_vars+=("JAVA_HOME=${JAVA_HOME:-}") fi local start start=$(date +%s) local exit_code=0 # macOS note: `timeout` is a GNU coreutils command not available on macOS by default # (requires `brew install coreutils`). On macOS, omit `timeout` from the env -i call # or guard it: command -v timeout &>/dev/null && TIMEOUT_CMD="timeout 120" || TIMEOUT_CMD="" env -i "${env_vars[@]}" \ timeout 120 \ ./connectors/"$connector"/run.sh \ > "$TMP_DIR/${connector}_${auth_type}.log" 2>&1 \ || exit_code=$? local duration=$(( $(date +%s) - start )) if [[ $exit_code -eq 0 ]]; then record_result "$connector" "$auth_type" "PASS" "$duration" "" log "[pass] $connector / $auth_type → PASS (${duration}s)" else record_result "$connector" "$auth_type" "FAIL" "$duration" "exit=$exit_code" log "[fail] $connector / $auth_type → FAIL (${duration}s)" # Show last 20 lines of log on failure tail -20 "$TMP_DIR/${connector}_${auth_type}.log" | while read -r line; do log " $line" done fi } ``` --- ## Parallel Execution Run multiple auth types concurrently by forking one subshell per auth type: ```bash if [[ $PARALLEL -eq 1 ]]; then IFS=',' read -ra par_auth_list <<< "$PARALLEL_AUTHS" pids=() for _par_auth in "${par_auth_list[@]}"; do _par_result_file="$TMP_DIR/results_${_par_auth}.tsv" touch "$_par_result_file" ( RESULTS_FILE="$_par_result_file" for connector in $ALL_CONNECTORS; do [[ -n "$FILTER_APP" ]] && [[ "$connector" != "$FILTER_APP" ]] && continue supports_auth "$connector" "$_par_auth" || continue run_connector_auth "$connector" "$_par_auth" done ) & pids+=($!) done # Wait for all parallel jobs set +e for _pid in "${pids[@]}"; do wait "$_pid"; done set -e # Merge result files for _par_auth in "${par_auth_list[@]}"; do _f="$TMP_DIR/results_${_par_auth}.tsv" [[ -f "$_f" ]] && cat "$_f" >> "$RESULTS_FILE" done fi ``` > **Why subshells, not background processes?** Each auth type modifies `RESULTS_FILE` and counters. Subshells get their own copies; variable changes in subshells don't propagate to the parent. After `wait`, merge the per-auth result files into the main `RESULTS_FILE`, then recount from the merged file. --- ## Usage Flags | Flag | Purpose | Example | |------|---------|---------| | `--env <name>` | Load credentials from `.env_<name>` | `--env demo` | | `--app <name>` | Test one connector only | `--app python-sdk` | | `--auth <type>` | Test one auth type only | `--auth oauth_m2m` | | `--parallel` | Run `PARALLEL_AUTHS` concurrently | `--parallel` | | `--parallel-auths <list>` | Comma-separated auth types to parallelize | `--parallel-auths pat,oauth_m2m` | | `--skip-build` | Skip build phase (re-run after fix) | `--skip-build` | **Browser tests (U2M):** ```bash PWAF_RUN_BROWSER_TESTS=1 ./tests/run_all_tests.sh --env demo --auth u2m_custom_oauth_app ``` --- ## Output Prefix Convention Each connector's run script should prefix its output lines with its connector name for easy log parsing: ```bash # In connectors/python-sdk/run.sh PREFIX="[pysdk]" echo "$PREFIX Connecting with auth: $APP_AUTH_TYPE" echo "$PREFIX PASS: tables.get returned samples.nyctaxi.trips" ``` The test runner captures this output per test. On failure, it shows the last 20 lines to help diagnose the issue. --- ## Pass/Fail/Skip Indicators | Indicator | Meaning | |-----------|---------| | `PASS` | Connector exited 0 — all operations succeeded | | `FAIL` | Connector exited non-zero — check log in `tmp/` | | `SKIP (build-failed)` | Build phase failed for this connector — fix build first | | `SKIP (browser-disabled)` | U2M browser test not run — set `PWAF_RUN_BROWSER_TESTS=1` | | `SKIP (no-token)` | Required env var not set in `.env` file | --- ## Report Generation After all tests complete, generate a Markdown report: ```bash generate_report() { local report_file="reports/test_report_${ENV_NAME}_$(date +%Y%m%dT%H%M%S).md" mkdir -p reports { echo "# PWAF Connector Test Report" echo "" echo "**Date:** $(date '+%Y-%m-%d %H:%M:%S')"