smoke-test

name	smoke-test
description	Start the Registry Server via docker compose and run a suite of curl-based smoke tests covering system, read-only MCP API, admin API, entry lifecycle, and OAuth/auth enforcement scenarios.
allowed-tools	Bash, Read
argument-hint	[keep-up]

Smoke Tests — Registry Server

Starts the Registry Server stack with docker compose, waits for it to be ready, then executes a suite of curl smoke tests organised as Gherkin scenarios. Pass keep-up as an argument to leave the stack running after the tests complete (useful for interactive exploration).

Embedded Scenarios

Feature: Registry Server smoke tests

  Background:
    Given the Registry Server is running at http://localhost:8080
    And the "default" registry is seeded from the upstream-registry.json file source

  # ── System endpoints ──────────────────────────────────────────────────────

  Scenario: Health check
    When I GET /health
    Then the response status is 200
    And the body contains "healthy"

  Scenario: Readiness check
    When I GET /readiness
    Then the response status is 200
    And the body contains "ready"

  Scenario: Version endpoint
    When I GET /version
    Then the response status is 200
    And the body contains "version"

  Scenario: OpenAPI spec
    When I GET /openapi.json
    Then the response status is 200
    And the body contains "openapi"

  # ── MCP Registry v0.1 — read-only ─────────────────────────────────────────

  Scenario: List servers in default registry
    When I GET /registry/default/v0.1/servers
    Then the response status is 200
    And the body contains "servers"

  Scenario: List servers with search filter
    When I GET /registry/default/v0.1/servers?search=mysql
    Then the response status is 200
    And the body contains "servers"

  Scenario: List servers with limit
    When I GET /registry/default/v0.1/servers?limit=2
    Then the response status is 200
    And the count of servers is at most 2

  Scenario: List servers filtered to latest version only
    When I GET /registry/default/v0.1/servers?version=latest
    Then the response status is 200
    And the body contains "servers"

  Scenario: List versions for a known server
    When I GET /registry/default/v0.1/servers/io.github.stacklok%2Fadb-mysql-mcp-server/versions
    Then the response status is 200
    And the body contains "servers"

  Scenario: Get a specific server version (latest)
    When I GET /registry/default/v0.1/servers/io.github.stacklok%2Fadb-mysql-mcp-server/versions/latest
    Then the response status is 200
    And the body contains "io.github.stacklok/adb-mysql-mcp-server"

  Scenario: Unknown registry returns 404
    When I GET /registry/nonexistent-registry/v0.1/servers
    Then the response status is 404

  Scenario: Unknown server version returns 404
    When I GET /registry/default/v0.1/servers/com.example%2Fdoes-not-exist/versions/1.0.0
    Then the response status is 404

  # ── Admin API v1 — registries ─────────────────────────────────────────────

  Scenario: List registries
    When I GET /v1/registries
    Then the response status is 200
    And the body contains "registries"

  Scenario: Get the default registry by name
    When I GET /v1/registries/default
    Then the response status is 200
    And the body contains "default"

  Scenario: Get a nonexistent registry returns 404
    When I GET /v1/registries/does-not-exist
    Then the response status is 404

  Scenario: Create a new registry via PUT
    When I PUT /v1/registries/test-registry with an empty source list
    Then the response status is 201
    And the body contains "test-registry"

  Scenario: Update an existing registry via PUT
    When I PUT /v1/registries/test-registry again with a description
    Then the response status is 200

  Scenario: Delete an API-created registry
    When I DELETE /v1/registries/test-registry
    Then the response status is 204

  Scenario: Delete a nonexistent registry returns 404
    When I DELETE /v1/registries/does-not-exist
    Then the response status is 404

  # ── Admin API v1 — sources ────────────────────────────────────────────────

  Scenario: List sources
    When I GET /v1/sources
    Then the response status is 200
    And the body contains "sources"

  Scenario: Get a known source by name
    When I GET /v1/sources/local-file
    Then the response status is 200
    And the body contains "local-file"

  Scenario: Get a nonexistent source returns 404
    When I GET /v1/sources/does-not-exist
    Then the response status is 404

  Scenario: Create a managed source
    When I PUT /v1/sources/managed-test with body {"managed":{}}
    Then the response status is 201
    And the body contains "managed-test"

  Scenario: Delete the managed source
    When I DELETE /v1/sources/managed-test
    Then the response status is 204

  # ── Entry lifecycle — publish and delete ──────────────────────────────────

  Scenario: Publish a server version
    Given a managed source exists
    When I POST /v1/entries with a server payload
    Then the response status is 201
    And the body contains the server name

  Scenario: Publish the same server version again returns 409
    When I POST /v1/entries with the same server payload
    Then the response status is 409

  Scenario: Publish a skill version
    When I POST /v1/entries with a skill payload
    Then the response status is 201
    And the body contains the skill name

  Scenario: Delete the published server version
    When I DELETE /v1/entries/server/com.example%2Ftest-server/versions/1.0.0
    Then the response status is 204

  Scenario: Delete a nonexistent entry returns 404
    When I DELETE /v1/entries/server/com.example%2Fnope/versions/9.9.9
    Then the response status is 404

  Scenario: Publish with both server and skill in body returns 400
    When I POST /v1/entries with both server and skill fields set
    Then the response status is 400

  Scenario: Publish with neither server nor skill in body returns 400
    When I POST /v1/entries with an empty payload
    Then the response status is 400

  # ── Claims update on edit ──────────────────────────────────────────────────

  Scenario: Registry claims are updated on PUT
    When I PUT /v1/registries/claims-test with claims {"org":"acme"}
    Then the response status is 201
    And the body contains "acme"
    When I PUT /v1/registries/claims-test with claims {"org":"contoso"}
    Then the response status is 200
    And the body contains "contoso"
    When I GET /v1/registries/claims-test
    Then the body contains "contoso"
    And the body does not contain "acme"

  Scenario: Source claims are updated on PUT
    When I PUT /v1/sources/claims-test with file-data and claims {"org":"acme"}
    Then the response status is 201
    When I GET /v1/sources/claims-test
    Then the body contains "acme"
    When I PUT /v1/sources/claims-test with file-data and claims {"org":"contoso"}
    Then the response status is 200
    When I GET /v1/sources/claims-test
    Then the body contains "contoso"
    And the body does not contain "acme"

  Scenario: Entry claims are updated via PUT
    When I PUT /v1/entries/skill/test-skill/claims with {"claims":{"team":"eng"}}
    Then the response status is 204

  # ── Managed source limit ──────────────────────────────────────────────────

  Scenario: Second managed source is rejected with 409
    Given the managed-test source still exists
    When I PUT /v1/sources/second-managed with body {"managed":{}}
    Then the response status is 409
    And the body contains "at most one managed source is allowed"

  # ── List completeness ─────────────────────────────────────────────────────

  Scenario: All created sources appear in list
    Given three file-data sources are created: smoke-src-a, smoke-src-b, smoke-src-c
    When I GET /v1/sources
    Then the response status is 200
    And the body contains "smoke-src-a"
    And the body contains "smoke-src-b"
    And the body contains "smoke-src-c"

  Scenario: All created registries appear in list
    Given three registries are created referencing the file-data sources
    When I GET /v1/registries
    Then the response status is 200
    And the body contains "smoke-reg-a"
    And the body contains "smoke-reg-b"
    And the body contains "smoke-reg-c"

  # ── OAuth / auth enforcement ───────────────────────────────────────────────
  # The stack is restarted with auth.mode: oauth before these scenarios run.
  # No real OIDC provider is needed: "missing token" and "malformed token"
  # cases are rejected before JWKS is consulted.

  Scenario: Public paths are accessible without a token in OAuth mode
    Given the server is restarted in OAuth mode
    When I GET /health
    Then the response status is 200
    When I GET /readiness
    Then the response status is 200
    When I GET /version
    Then the response status is 200
    When I GET /openapi.json
    Then the response status is 200

  Scenario: OAuth protected-resource metadata is publicly accessible
    When I GET /.well-known/oauth-protected-resource
    Then the response status is 200
    And the body contains "authorization_servers"

  Scenario: MCP list-servers requires a token in OAuth mode
    When I GET /registry/default/v0.1/servers without an Authorization header
    Then the response status is 401
    And the response includes a WWW-Authenticate header with Bearer scheme
    And the WWW-Authenticate header contains resource_metadata

  Scenario: Admin registries endpoint requires a token
    When I GET /v1/registries without an Authorization header
    Then the response status is 401

  Scenario: Admin sources endpoint requires a token
    When I GET /v1/sources without an Authorization header
    Then the response status is 401

  Scenario: Malformed Bearer token returns 401 invalid_token
    When I GET /registry/default/v0.1/servers with Authorization: Bearer not-a-real-jwt
    Then the response status is 401
    And the WWW-Authenticate header contains error="invalid_token"

---

Steps

1. Start the stack

BASE_URL="http://localhost:8080"
PROJECT="thv-smoke-test"
COMPOSE_FILE="docker-compose.smoke-test.yaml"
PASS=0
FAIL=0

# Build the image only when it does not already exist locally.
# Separating build from up avoids a Docker Hub metadata fetch that can hang
# when the base image is pinned by digest (the default for this project).
if ! docker image inspect thv-smoke-test-registry-api:latest > /dev/null 2>&1; then
  echo "=== Building Registry Server image ==="
  # Use the same project name so the image is tagged thv-smoke-test-registry-api:latest,
  # matching what docker-compose.smoke-test.yaml expects.
  docker compose --project-name "$PROJECT" build 2>&1 || {
    echo "ERROR: docker compose build failed"
    exit 1
  }
fi

echo "=== Starting Registry Server stack ==="
docker compose --project-name "$PROJECT" -f "$COMPOSE_FILE" up --detach --wait 2>&1 || {
  echo "ERROR: docker compose failed to start"
  docker compose --project-name "$PROJECT" -f "$COMPOSE_FILE" logs
  exit 1
}
echo "Stack is up."

2. Wait for readiness

# docker compose --wait already blocks until all healthchecks pass, so this
# step is a lightweight confirmation rather than an active poll.
BASE_URL="http://localhost:8080"
echo "=== Confirming /readiness ==="
curl -sf "$BASE_URL/readiness" > /dev/null 2>&1 && echo "Server is ready." || {
  echo "ERROR: /readiness did not respond"
  docker compose --project-name thv-smoke-test -f docker-compose.smoke-test.yaml logs registry-api
  exit 1
}

3. Define test helper and run scenarios

BASE_URL="http://localhost:8080"
PASS=0
FAIL=0

# ─── helper ───────────────────────────────────────────────────────────────────
# check DESCRIPTION EXPECTED_STATUS ACTUAL_STATUS BODY [GREP_PATTERN]
check() {
  local desc="$1" expected="$2" actual="$3" body="$4" pattern="${5:-}"
  local ok=true

  # Accept a range like "4xx" or "5xx"
  if echo "$expected" | grep -qE '^[45]xx$'; then
    local prefix="${expected:0:1}"
    if ! echo "$actual" | grep -qE "^${prefix}[0-9]{2}$"; then
      ok=false
    fi
  elif [ "$actual" != "$expected" ]; then
    ok=false
  fi

  if $ok && [ -n "$pattern" ]; then
    if ! echo "$body" | grep -q "$pattern"; then
      ok=false
    fi
  fi

  if $ok; then
    echo "  ✓ $desc"
    PASS=$((PASS + 1))
  else
    echo "  ✗ $desc  [expected HTTP $expected, got $actual]"
    if [ -n "$pattern" ] && ! echo "$body" | grep -q "$pattern"; then
      echo "    (body did not contain: $pattern)"
      echo "    body: $(echo "$body" | head -3)"
    fi
    FAIL=$((FAIL + 1))
  fi
}

curl_get() { curl -s -o /tmp/thv_body -w "%{http_code}" "$BASE_URL$1"; }
curl_post() { curl -s -o /tmp/thv_body -w "%{http_code}" -X POST -H "Content-Type: application/json" -d "$2" "$BASE_URL$1"; }
curl_put()  { curl -s -o /tmp/thv_body -w "%{http_code}" -X PUT  -H "Content-Type: application/json" -d "$2" "$BASE_URL$1"; }
curl_del()  { curl -s -o /tmp/thv_body -w "%{http_code}" -X DELETE "$BASE_URL$1"; }
body()      { cat /tmp/thv_body; }

# ─── System endpoints ─────────────────────────────────────────────────────────
echo ""
echo "── System endpoints ──"

SC=$(curl_get /health);    check "GET /health returns 200 with healthy" 200 "$SC" "$(body)" "healthy"
SC=$(curl_get /readiness); check "GET /readiness returns 200 with ready" 200 "$SC" "$(body)" "ready"
SC=$(curl_get /version);   check "GET /version returns 200 with version field" 200 "$SC" "$(body)" "version"
SC=$(curl_get /openapi.json); check "GET /openapi.json returns 200 with openapi field" 200 "$SC" "$(body)" "openapi"

# ─── MCP Registry v0.1 ───────────────────────────────────────────────────────
echo ""
echo "── MCP Registry v0.1 ──"

SC=$(curl_get /registry/default/v0.1/servers)
check "GET /registry/default/v0.1/servers returns 200" 200 "$SC" "$(body)" "servers"

SC=$(curl_get "/registry/default/v0.1/servers?search=mysql")
check "GET .../servers?search=mysql returns 200" 200 "$SC" "$(body)" "servers"

SC=$(curl_get "/registry/default/v0.1/servers?limit=2")
check "GET .../servers?limit=2 returns 200" 200 "$SC" "$(body)" "servers"
SERVER_COUNT=$(body | grep -o '"count":[0-9]*' | grep -o '[0-9]*' | head -1)
SERVER_COUNT="${SERVER_COUNT:-0}"
if [ "$SERVER_COUNT" -le 2 ]; then
  echo "  ✓ limit=2 returned at most 2 servers (count=$SERVER_COUNT)"
  PASS=$((PASS + 1))
else
  echo "  ✗ limit=2 returned more than 2 servers (count=$SERVER_COUNT)"
  FAIL=$((FAIL + 1))
fi

SC=$(curl_get "/registry/default/v0.1/servers?version=latest")
check "GET .../servers?version=latest returns 200" 200 "$SC" "$(body)" "servers"

SC=$(curl_get "/registry/default/v0.1/servers/io.github.stacklok%2Fadb-mysql-mcp-server/versions")
check "GET .../servers/{name}/versions returns 200" 200 "$SC" "$(body)" "servers"

SC=$(curl_get "/registry/default/v0.1/servers/io.github.stacklok%2Fadb-mysql-mcp-server/versions/latest")
check "GET .../servers/{name}/versions/latest returns 200 with name" 200 "$SC" "$(body)" "adb-mysql-mcp-server"

SC=$(curl_get /registry/nonexistent-registry/v0.1/servers)
check "GET /registry/nonexistent-registry/... returns 404" 404 "$SC" "$(body)"

SC=$(curl_get "/registry/default/v0.1/servers/com.example%2Fdoes-not-exist/versions/1.0.0")
check "GET unknown server version returns 404" 404 "$SC" "$(body)"

# ─── Admin v1 — registries ────────────────────────────────────────────────────
echo ""
echo "── Admin v1 — registries ──"

SC=$(curl_get /v1/registries)
check "GET /v1/registries returns 200" 200 "$SC" "$(body)" "registries"

SC=$(curl_get /v1/registries/default)
check "GET /v1/registries/default returns 200" 200 "$SC" "$(body)" "default"

SC=$(curl_get /v1/registries/does-not-exist)
check "GET /v1/registries/does-not-exist returns 404" 404 "$SC" "$(body)"

SC=$(curl_put /v1/registries/test-registry '{"sources":["local-file"]}')
check "PUT /v1/registries/test-registry (create) returns 201" 201 "$SC" "$(body)" "test-registry"