pwaf-connector-testing

name	pwaf-connector-testing
description	How to build and use a PWAF connector test runner (tests/run_all_tests.sh). Covers env isolation, auth types, single connector or single auth mode, parallel execution, browser tests, and report generation.

PWAF Connector Testing

Use this skill when building or using a test runner for PWAF connector projects. The pattern here can be applied to any new connector project to test all connectors across all auth types.

---

Overview

A PWAF connector test runner (tests/run_all_tests.sh) validates every connector (REST API, Python SDK, SQL drivers, Go, Java, Node.js) across all supported auth types (PAT, OAuth M2M, OAuth U2M), with proper environment isolation for each test.

Key design principles:

1. Each test runs in a clean env -i subprocess — never inherit full shell environment
2. Each auth type has its own set of required variables — never mix them
3. Browser tests (U2M) are opt-in — require PWAF_RUN_BROWSER_TESTS=1
4. Build failures skip tests — don't fail the run, just mark as skipped
5. Results written to a report file for review

---

Environment File

The test runner reads credentials from a .env file (passed with --env <name>). The file is located at .env_<name> in the project root.

# .env_demo — example
DATABRICKS_HOST=https://myworkspace.cloud.databricks.com
DATABRICKS_TOKEN=dapiXXXXXXXXXXXXXXXX
DATABRICKS_CLIENT_ID=11111111-2222-3333-4444-555555555555
DATABRICKS_CLIENT_SECRET=doceXXXXXXXXXXXXXXXX
DATABRICKS_U2M_CLIENT_ID=66666666-7777-8888-9999-000000000000
DATABRICKS_U2M_CLIENT_SECRET=doceYYYYYYYYYYYYYYYY
DATABRICKS_REDIRECT_URI=http://localhost:8080/callback
DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/abcdef123456
DATABRICKS_WAREHOUSE_ID=abcdef123456
CATALOG=my_catalog

---

Auth Type → Required Variables

Each auth type is tested in its own env -i subprocess with only these variables:

Auth type	Required env vars
pat	DATABRICKS_HOST, DATABRICKS_TOKEN
oauth_m2m	DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET
u2m_custom_oauth_app	DATABRICKS_HOST, DATABRICKS_U2M_CLIENT_ID, DATABRICKS_U2M_CLIENT_SECRET (if confidential), DATABRICKS_REDIRECT_URI
u2m_token_env	DATABRICKS_HOST, DATABRICKS_ACCESS_TOKEN (or DATABRICKS_TOKEN)

Always include in the env -i base: PATH, HOME, USER, LANG.

Never include DATABRICKS_AUTH_TYPE in any test subprocess. The Python SDK, Go SDK, and Java SDK all read it internally. Setting it to application-level values like "oauth_m2m" breaks credential resolution. Use APP_AUTH_TYPE=<value> as the application-level selector passed to the connector.

---

Script Structure

#!/usr/bin/env bash
set -euo pipefail

# === Defaults ===
ENV_NAME="demo"
FILTER_APP=""
FILTER_AUTH=""
PARALLEL=0
PARALLEL_AUTHS="pat,oauth_m2m"
SKIP_BUILD=0
BROWSER_TESTS="${PWAF_RUN_BROWSER_TESTS:-0}"

# === Argument parsing ===
while [[ $# -gt 0 ]]; do
    case "$1" in
        --env)       ENV_NAME="$2";       shift 2 ;;
        --app)       FILTER_APP="$2";     shift 2 ;;
        --auth)      FILTER_AUTH="$2";    shift 2 ;;
        --parallel)  PARALLEL=1;          shift 1 ;;
        --parallel-auths) PARALLEL_AUTHS="$2"; shift 2 ;;
        --skip-build) SKIP_BUILD=1;       shift 1 ;;
        *) echo "Unknown option: $1"; exit 1 ;;
    esac
done

ENV_FILE=".env_${ENV_NAME}"
source "$ENV_FILE"

---

Build Phase

Build each connector before running tests. Record build failures — tests for failed connectors are automatically skipped.

build_connector() {
    local name="$1"
    local dir="connectors/$name"

    case "$name" in
        go-sdk|go-sql-driver)
            (cd "$dir" && go build ./...) ;;
        java-sdk|java-jdbc)
            (cd "$dir" && mvn -q package -DskipTests) ;;
        nodejs-sql-driver)
            (cd "$dir" && npm install && npm run build) ;;
        python-*|rest-api)
            true ;;  # No build step for Python
    esac
}

---

Test Execution with env -i

Each connector test runs in a clean subprocess. The run_connector_auth function:

1. Checks if this connector supports this auth type
2. Skips browser tests unless PWAF_RUN_BROWSER_TESTS=1
3. Builds the env -i environment with only auth-specific vars
4. Runs the connector with a timeout
5. Records PASS/FAIL/SKIP

run_connector_auth() {
    local connector="$1"
    local auth_type="$2"

    # Skip if build failed
    if build_failed "$connector"; then
        record_result "$connector" "$auth_type" "SKIP" 0 "build-failed"
        return
    fi

    # Skip browser tests unless enabled
    if is_browser_auth "$auth_type" && [[ "$BROWSER_TESTS" != "1" ]]; then
        record_result "$connector" "$auth_type" "SKIP" 0 "browser-disabled"
        return
    fi

    # Build env -i vars for this auth type
    local env_vars=(
        "PATH=$PATH"
        "HOME=$HOME"
        "USER=${USER:-}"
        "LANG=${LANG:-en_US.UTF-8}"
        "DATABRICKS_HOST=$DATABRICKS_HOST"
        "APP_AUTH_TYPE=$auth_type"
    )

    case "$auth_type" in
        pat)
            env_vars+=("DATABRICKS_TOKEN=$DATABRICKS_TOKEN") ;;
        oauth_m2m)
            env_vars+=(
                "DATABRICKS_CLIENT_ID=$DATABRICKS_CLIENT_ID"
                "DATABRICKS_CLIENT_SECRET=$DATABRICKS_CLIENT_SECRET"
            ) ;;
        u2m_custom_oauth_app)
            env_vars+=(
                "DATABRICKS_U2M_CLIENT_ID=$DATABRICKS_U2M_CLIENT_ID"
                "DATABRICKS_U2M_CLIENT_SECRET=${DATABRICKS_U2M_CLIENT_SECRET:-}"
                "DATABRICKS_REDIRECT_URI=${DATABRICKS_REDIRECT_URI:-http://localhost:8080/callback}"
            ) ;;
        u2m_token_env)
            env_vars+=("DATABRICKS_ACCESS_TOKEN=$DATABRICKS_ACCESS_TOKEN") ;;
    esac

    # Add SQL-specific vars if connector needs them
    if connector_needs_sql "$connector"; then
        env_vars+=(
            "DATABRICKS_HTTP_PATH=$DATABRICKS_HTTP_PATH"
            "DATABRICKS_WAREHOUSE_ID=${DATABRICKS_WAREHOUSE_ID:-}"
        )
    fi

    # Add Java-specific vars
    if [[ "$connector" == java-* ]]; then
        env_vars+=("JAVA_HOME=${JAVA_HOME:-}")
    fi

    local start
    start=$(date +%s)
    local exit_code=0

    # macOS note: `timeout` is a GNU coreutils command not available on macOS by default
    # (requires `brew install coreutils`). On macOS, omit `timeout` from the env -i call
    # or guard it: command -v timeout &>/dev/null && TIMEOUT_CMD="timeout 120" || TIMEOUT_CMD=""
    env -i "${env_vars[@]}" \
        timeout 120 \
        ./connectors/"$connector"/run.sh \
        > "$TMP_DIR/${connector}_${auth_type}.log" 2>&1 \
        || exit_code=$?

    local duration=$(( $(date +%s) - start ))

    if [[ $exit_code -eq 0 ]]; then
        record_result "$connector" "$auth_type" "PASS" "$duration" ""
        log "[pass]   $connector / $auth_type → PASS (${duration}s)"
    else
        record_result "$connector" "$auth_type" "FAIL" "$duration" "exit=$exit_code"
        log "[fail]   $connector / $auth_type → FAIL (${duration}s)"
        # Show last 20 lines of log on failure
        tail -20 "$TMP_DIR/${connector}_${auth_type}.log" | while read -r line; do
            log "         $line"
        done
    fi
}

---

Parallel Execution

Run multiple auth types concurrently by forking one subshell per auth type:

if [[ $PARALLEL -eq 1 ]]; then
    IFS=',' read -ra par_auth_list <<< "$PARALLEL_AUTHS"
    pids=()

    for _par_auth in "${par_auth_list[@]}"; do
        _par_result_file="$TMP_DIR/results_${_par_auth}.tsv"
        touch "$_par_result_file"
        (
            RESULTS_FILE="$_par_result_file"
            for connector in $ALL_CONNECTORS; do
                [[ -n "$FILTER_APP" ]] && [[ "$connector" != "$FILTER_APP" ]] && continue
                supports_auth "$connector" "$_par_auth" || continue
                run_connector_auth "$connector" "$_par_auth"
            done
        ) &
        pids+=($!)
    done

    # Wait for all parallel jobs
    set +e
    for _pid in "${pids[@]}"; do wait "$_pid"; done
    set -e

    # Merge result files
    for _par_auth in "${par_auth_list[@]}"; do
        _f="$TMP_DIR/results_${_par_auth}.tsv"
        [[ -f "$_f" ]] && cat "$_f" >> "$RESULTS_FILE"
    done
fi

Why subshells, not background processes? Each auth type modifies RESULTS_FILE and counters. Subshells get their own copies; variable changes in subshells don't propagate to the parent. After wait, merge the per-auth result files into the main RESULTS_FILE, then recount from the merged file.

---

Usage Flags

Flag	Purpose	Example
--env <name>	Load credentials from .env_<name>	--env demo
--app <name>	Test one connector only	--app python-sdk
--auth <type>	Test one auth type only	--auth oauth_m2m
--parallel	Run PARALLEL_AUTHS concurrently	--parallel
--parallel-auths <list>	Comma-separated auth types to parallelize	--parallel-auths pat,oauth_m2m
--skip-build	Skip build phase (re-run after fix)	--skip-build

Browser tests (U2M):

PWAF_RUN_BROWSER_TESTS=1 ./tests/run_all_tests.sh --env demo --auth u2m_custom_oauth_app

---

Output Prefix Convention

Each connector's run script should prefix its output lines with its connector name for easy log parsing:

# In connectors/python-sdk/run.sh
PREFIX="[pysdk]"
echo "$PREFIX Connecting with auth: $APP_AUTH_TYPE"
echo "$PREFIX PASS: tables.get returned samples.nyctaxi.trips"

The test runner captures this output per test. On failure, it shows the last 20 lines to help diagnose the issue.

---

Pass/Fail/Skip Indicators

Indicator	Meaning
PASS	Connector exited 0 — all operations succeeded
FAIL	Connector exited non-zero — check log in tmp/
SKIP (build-failed)	Build phase failed for this connector — fix build first
SKIP (browser-disabled)	U2M browser test not run — set PWAF_RUN_BROWSER_TESTS=1
SKIP (no-token)	Required env var not set in .env file

---

Report Generation

After all tests complete, generate a Markdown report:

generate_report() {
    local report_file="reports/test_report_${ENV_NAME}_$(date +%Y%m%dT%H%M%S).md"
    mkdir -p reports

    {
        echo "# PWAF Connector Test Report"
        echo ""
        echo "**Date:** $(date '+%Y-%m-%d %H:%M:%S')"
        echo "**Env:** $ENV_NAME"
        echo "**Workspace:** $DATABRICKS_HOST"
        echo ""
        echo "## Results Summary"
        echo ""
        echo "| Connector | Auth Type | Result | Duration |"
        echo "|-----------|-----------|--------|----------|"
        while IFS='|' read -r _connector _auth _result _duration _note; do
            echo "| $_connector | $_auth | $_result | ${_duration}s |"
        done < "$RESULTS_FILE"
        echo ""
        echo "**Passed:** $PASSED / $TOTAL"
        echo "**Failed:** $FAILED"
        echo "**Skipped:** $((SKIPPED_BROWSER + SKIPPED_TOKEN))"
    } > "$report_file"

    echo "Report: $report_file"
}

Reports are saved to reports/ and named test_report_<env>_<timestamp>.md.

---

Connector run.sh Contract

Each connector needs a run.sh that:

• Returns exit code 0 on success, non-zero on failure
• Reads auth type from APP_AUTH_TYPE
• Uses only the env vars provided (no hardcoded credentials)
• Prints progress lines prefixed with connector name

#!/usr/bin/env bash
# connectors/python-sdk/run.sh
set -euo pipefail
PREFIX="[pysdk]"
AUTH="${APP_AUTH_TYPE:-pat}"
echo "$PREFIX Testing auth: $AUTH"
python3 main.py
echo "$PREFIX DONE"

---

Quick Start: New Project

To build the test runner for a new PWAF connector project:

1. Create tests/run_all_tests.sh using the structure above
2. Create connectors/<name>/run.sh for each connector
3. Create .env_<name> with credentials
4. Run: ./tests/run_all_tests.sh --env <name>
5. Run browser tests: PWAF_RUN_BROWSER_TESTS=1 ./tests/run_all_tests.sh --env <name> --auth u2m_custom_oauth_app

For a single connector across all auth types:

./tests/run_all_tests.sh --env demo --app python-sdk

For all connectors with one auth type:

./tests/run_all_tests.sh --env demo --auth oauth_m2m