# Pwaf Connector Testing

> How to build and use a PWAF connector test runner (tests/run_all_tests.sh). Covers env isolation, auth types, single connector or single auth mode, parallel execution, browser tests, and report generation.

- Skill: `databricks-solutions/pwaf-connector-testing` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add databricks-solutions/pwaf-connector-testing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/databricks-solutions/pwaf-connector-testing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: databricks-solutions (https://skillmd.com/u/databricks-solutions)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/databricks-solutions/pwaf-connector-testing

---


<!-- skill-version: 1.0.0 -->

# PWAF Connector Testing

Use this skill when building or using a test runner for PWAF connector projects. The pattern here can be applied to any new connector project to test all connectors across all auth types.

---

## Overview

A PWAF connector test runner (`tests/run_all_tests.sh`) validates every connector (REST API, Python SDK, SQL drivers, Go, Java, Node.js) across all supported auth types (PAT, OAuth M2M, OAuth U2M), with proper environment isolation for each test.

Key design principles:
1. **Each test runs in a clean `env -i` subprocess** — never inherit full shell environment
2. **Each auth type has its own set of required variables** — never mix them
3. **Browser tests (U2M) are opt-in** — require `PWAF_RUN_BROWSER_TESTS=1`
4. **Build failures skip tests** — don't fail the run, just mark as skipped
5. **Results written to a report file** for review

---

## Environment File

The test runner reads credentials from a `.env` file (passed with `--env <name>`). The file is located at `.env_<name>` in the project root.

```bash
# .env_demo — example
DATABRICKS_HOST=https://myworkspace.cloud.databricks.com
DATABRICKS_TOKEN=dapiXXXXXXXXXXXXXXXX
DATABRICKS_CLIENT_ID=11111111-2222-3333-4444-555555555555
DATABRICKS_CLIENT_SECRET=doceXXXXXXXXXXXXXXXX
DATABRICKS_U2M_CLIENT_ID=66666666-7777-8888-9999-000000000000
DATABRICKS_U2M_CLIENT_SECRET=doceYYYYYYYYYYYYYYYY
DATABRICKS_REDIRECT_URI=http://localhost:8080/callback
DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/abcdef123456
DATABRICKS_WAREHOUSE_ID=abcdef123456
CATALOG=my_catalog
```

---

## Auth Type → Required Variables

Each auth type is tested in its own `env -i` subprocess with only these variables:

| Auth type | Required env vars |
|-----------|------------------|
| `pat` | `DATABRICKS_HOST`, `DATABRICKS_TOKEN` |
| `oauth_m2m` | `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET` |
| `u2m_custom_oauth_app` | `DATABRICKS_HOST`, `DATABRICKS_U2M_CLIENT_ID`, `DATABRICKS_U2M_CLIENT_SECRET` (if confidential), `DATABRICKS_REDIRECT_URI` |
| `u2m_token_env` | `DATABRICKS_HOST`, `DATABRICKS_ACCESS_TOKEN` (or `DATABRICKS_TOKEN`) |

Always include in the `env -i` base: `PATH`, `HOME`, `USER`, `LANG`.

> **Never include `DATABRICKS_AUTH_TYPE` in any test subprocess.** The Python SDK, Go SDK, and Java SDK all read it internally. Setting it to application-level values like `"oauth_m2m"` breaks credential resolution.
> Use `APP_AUTH_TYPE=<value>` as the application-level selector passed to the connector.

---

## Script Structure

```bash
#!/usr/bin/env bash
set -euo pipefail

# === Defaults ===
ENV_NAME="demo"
FILTER_APP=""
FILTER_AUTH=""
PARALLEL=0
PARALLEL_AUTHS="pat,oauth_m2m"
SKIP_BUILD=0
BROWSER_TESTS="${PWAF_RUN_BROWSER_TESTS:-0}"

# === Argument parsing ===
while [[ $# -gt 0 ]]; do
    case "$1" in
        --env)       ENV_NAME="$2";       shift 2 ;;
        --app)       FILTER_APP="$2";     shift 2 ;;
        --auth)      FILTER_AUTH="$2";    shift 2 ;;
        --parallel)  PARALLEL=1;          shift 1 ;;
        --parallel-auths) PARALLEL_AUTHS="$2"; shift 2 ;;
        --skip-build) SKIP_BUILD=1;       shift 1 ;;
        *) echo "Unknown option: $1"; exit 1 ;;
    esac
done

ENV_FILE=".env_${ENV_NAME}"
source "$ENV_FILE"
```

---

## Build Phase

Build each connector before running tests. Record build failures — tests for failed connectors are automatically skipped.

```bash
build_connector() {
    local name="$1"
    local dir="connectors/$name"

    case "$name" in
        go-sdk|go-sql-driver)
            (cd "$dir" && go build ./...) ;;
        java-sdk|java-jdbc)
            (cd "$dir" && mvn -q package -DskipTests) ;;
        nodejs-sql-driver)
            (cd "$dir" && npm install && npm run build) ;;
        python-*|rest-api)
            true ;;  # No build step for Python
    esac
}
```

---

## Test Execution with env -i

Each connector test runs in a clean subprocess. The `run_connector_auth` function:
1. Checks if this connector supports this auth type
2. Skips browser tests unless `PWAF_RUN_BROWSER_TESTS=1`
3. Builds the `env -i` environment with only auth-specific vars
4. Runs the connector with a timeout
5. Records PASS/FAIL/SKIP

```bash
run_connector_auth() {
    local connector="$1"
    local auth_type="$2"

    # Skip if build failed
    if build_failed "$connector"; then
        record_result "$connector" "$auth_type" "SKIP" 0 "build-failed"
        return
    fi

    # Skip browser tests unless enabled
    if is_browser_auth "$auth_type" && [[ "$BROWSER_TESTS" != "1" ]]; then
        record_result "$connector" "$auth_type" "SKIP" 0 "browser-disabled"
        return
    fi

    # Build env -i vars for this auth type
    local env_vars=(
        "PATH=$PATH"
        "HOME=$HOME"
        "USER=${USER:-}"
        "LANG=${LANG:-en_US.UTF-8}"
        "DATABRICKS_HOST=$DATABRICKS_HOST"
        "APP_AUTH_TYPE=$auth_type"
    )

    case "$auth_type" in
        pat)
            env_vars+=("DATABRICKS_TOKEN=$DATABRICKS_TOKEN") ;;
        oauth_m2m)
            env_vars+=(
                "DATABRICKS_CLIENT_ID=$DATABRICKS_CLIENT_ID"
                "DATABRICKS_CLIENT_SECRET=$DATABRICKS_CLIENT_SECRET"
            ) ;;
        u2m_custom_oauth_app)
            env_vars+=(
                "DATABRICKS_U2M_CLIENT_ID=$DATABRICKS_U2M_CLIENT_ID"
                "DATABRICKS_U2M_CLIENT_SECRET=${DATABRICKS_U2M_CLIENT_SECRET:-}"
                "DATABRICKS_REDIRECT_URI=${DATABRICKS_REDIRECT_URI:-http://localhost:8080/callback}"
            ) ;;
        u2m_token_env)
            env_vars+=("DATABRICKS_ACCESS_TOKEN=$DATABRICKS_ACCESS_TOKEN") ;;
    esac

    # Add SQL-specific vars if connector needs them
    if connector_needs_sql "$connector"; then
        env_vars+=(
            "DATABRICKS_HTTP_PATH=$DATABRICKS_HTTP_PATH"
            "DATABRICKS_WAREHOUSE_ID=${DATABRICKS_WAREHOUSE_ID:-}"
        )
    fi

    # Add Java-specific vars
    if [[ "$connector" == java-* ]]; then
        env_vars+=("JAVA_HOME=${JAVA_HOME:-}")
    fi

    local start
    start=$(date +%s)
    local exit_code=0

    # macOS note: `timeout` is a GNU coreutils command not available on macOS by default
    # (requires `brew install coreutils`). On macOS, omit `timeout` from the env -i call
    # or guard it: command -v timeout &>/dev/null && TIMEOUT_CMD="timeout 120" || TIMEOUT_CMD=""
    env -i "${env_vars[@]}" \
        timeout 120 \
        ./connectors/"$connector"/run.sh \
        > "$TMP_DIR/${connector}_${auth_type}.log" 2>&1 \
        || exit_code=$?

    local duration=$(( $(date +%s) - start ))

    if [[ $exit_code -eq 0 ]]; then
        record_result "$connector" "$auth_type" "PASS" "$duration" ""
        log "[pass]   $connector / $auth_type → PASS (${duration}s)"
    else
        record_result "$connector" "$auth_type" "FAIL" "$duration" "exit=$exit_code"
        log "[fail]   $connector / $auth_type → FAIL (${duration}s)"
        # Show last 20 lines of log on failure
        tail -20 "$TMP_DIR/${connector}_${auth_type}.log" | while read -r line; do
            log "         $line"
        done
    fi
}
```

---

## Parallel Execution

Run multiple auth types concurrently by forking one subshell per auth type:

```bash
if [[ $PARALLEL -eq 1 ]]; then
    IFS=',' read -ra par_auth_list <<< "$PARALLEL_AUTHS"
    pids=()

    for _par_auth in "${par_auth_list[@]}"; do
        _par_result_file="$TMP_DIR/results_${_par_auth}.tsv"
        touch "$_par_result_file"
        (
            RESULTS_FILE="$_par_result_file"
            for connector in $ALL_CONNECTORS; do
                [[ -n "$FILTER_APP" ]] && [[ "$connector" != "$FILTER_APP" ]] && continue
                supports_auth "$connector" "$_par_auth" || continue
                run_connector_auth "$connector" "$_par_auth"
            done
        ) &
        pids+=($!)
    done

    # Wait for all parallel jobs
    set +e
    for _pid in "${pids[@]}"; do wait "$_pid"; done
    set -e

    # Merge result files
    for _par_auth in "${par_auth_list[@]}"; do
        _f="$TMP_DIR/results_${_par_auth}.tsv"
        [[ -f "$_f" ]] && cat "$_f" >> "$RESULTS_FILE"
    done
fi
```

> **Why subshells, not background processes?** Each auth type modifies `RESULTS_FILE` and counters. Subshells get their own copies; variable changes in subshells don't propagate to the parent. After `wait`, merge the per-auth result files into the main `RESULTS_FILE`, then recount from the merged file.

---

## Usage Flags

| Flag | Purpose | Example |
|------|---------|---------|
| `--env <name>` | Load credentials from `.env_<name>` | `--env demo` |
| `--app <name>` | Test one connector only | `--app python-sdk` |
| `--auth <type>` | Test one auth type only | `--auth oauth_m2m` |
| `--parallel` | Run `PARALLEL_AUTHS` concurrently | `--parallel` |
| `--parallel-auths <list>` | Comma-separated auth types to parallelize | `--parallel-auths pat,oauth_m2m` |
| `--skip-build` | Skip build phase (re-run after fix) | `--skip-build` |

**Browser tests (U2M):**
```bash
PWAF_RUN_BROWSER_TESTS=1 ./tests/run_all_tests.sh --env demo --auth u2m_custom_oauth_app
```

---

## Output Prefix Convention

Each connector's run script should prefix its output lines with its connector name for easy log parsing:

```bash
# In connectors/python-sdk/run.sh
PREFIX="[pysdk]"
echo "$PREFIX Connecting with auth: $APP_AUTH_TYPE"
echo "$PREFIX PASS: tables.get returned samples.nyctaxi.trips"
```

The test runner captures this output per test. On failure, it shows the last 20 lines to help diagnose the issue.

---

## Pass/Fail/Skip Indicators

| Indicator | Meaning |
|-----------|---------|
| `PASS` | Connector exited 0 — all operations succeeded |
| `FAIL` | Connector exited non-zero — check log in `tmp/` |
| `SKIP (build-failed)` | Build phase failed for this connector — fix build first |
| `SKIP (browser-disabled)` | U2M browser test not run — set `PWAF_RUN_BROWSER_TESTS=1` |
| `SKIP (no-token)` | Required env var not set in `.env` file |

---

## Report Generation

After all tests complete, generate a Markdown report:

```bash
generate_report() {
    local report_file="reports/test_report_${ENV_NAME}_$(date +%Y%m%dT%H%M%S).md"
    mkdir -p reports

    {
        echo "# PWAF Connector Test Report"
        echo ""
        echo "**Date:** $(date '+%Y-%m-%d %H:%M:%S')"
        echo "**Env:** $ENV_NAME"
        echo "**Workspace:** $DATABRICKS_HOST"
        echo ""
        echo "## Results Summary"
        echo ""
        echo "| Connector | Auth Type | Result | Duration |"
        echo "|-----------|-----------|--------|----------|"
        while IFS='|' read -r _connector _auth _result _duration _note; do
            echo "| $_connector | $_auth | $_result | ${_duration}s |"
        done < "$RESULTS_FILE"
        echo ""
        echo "**Passed:** $PASSED / $TOTAL"
        echo "**Failed:** $FAILED"
        echo "**Skipped:** $((SKIPPED_BROWSER + SKIPPED_TOKEN))"
    } > "$report_file"

    echo "Report: $report_file"
}
```

Reports are saved to `reports/` and named `test_report_<env>_<timestamp>.md`.

---

## Connector `run.sh` Contract

Each connector needs a `run.sh` that:
- Returns exit code `0` on success, non-zero on failure
- Reads auth type from `APP_AUTH_TYPE`
- Uses only the env vars provided (no hardcoded credentials)
- Prints progress lines prefixed with connector name

```bash
#!/usr/bin/env bash
# connectors/python-sdk/run.sh
set -euo pipefail
PREFIX="[pysdk]"
AUTH="${APP_AUTH_TYPE:-pat}"
echo "$PREFIX Testing auth: $AUTH"
python3 main.py
echo "$PREFIX DONE"
```

---

## Quick Start: New Project

To build the test runner for a new PWAF connector project:

1. Create `tests/run_all_tests.sh` using the structure above
2. Create `connectors/<name>/run.sh` for each connector
3. Create `.env_<name>` with credentials
4. Run: `./tests/run_all_tests.sh --env <name>`
5. Run browser tests: `PWAF_RUN_BROWSER_TESTS=1 ./tests/run_all_tests.sh --env <name> --auth u2m_custom_oauth_app`

For a single connector across all auth types:
```bash
./tests/run_all_tests.sh --env demo --app python-sdk
```

For all connectors with one auth type:
```bash
./tests/run_all_tests.sh --env demo --auth oauth_m2m
```