| name | integration-testing |
| description | Guidance for writing and running Python/pytest integration tests against APIM, App Gateway, and AI services in ai-hub-tracking. |
Integration Testing Skills
Use this skill profile when creating, modifying, or debugging integration tests in this repo.
Use When
- Writing new pytest suites under
tests/integration/tests/
- Modifying the shared integration runtime in
tests/integration/src/ai_hub_integration/
- Debugging CI or local failures in the secure-tunnel integration workflow
- Extending
run-tests.py, run-tests.sh, or suite alias/group behavior
Do Not Use When
- Changing APIM policy behavior without updating tests (use API Management first, then come here)
- Modifying Terraform infrastructure without touching test execution or assertions (use IaC Coder)
- Working on Azure AI Evaluation scoring logic only (use AI Evaluation)
Input Contract
Required context before making test changes:
- Which tenant(s), API paths, and backend behaviors are under test
- Whether the scenario requires App Gateway, Key Vault fallback, or the secure tunnel
- Any feature flags that gate the behavior (
document_intelligence_enabled, key rotation, tenant capabilities)
Output Contract
Every test change should deliver:
- Pytest functions with explicit, behavior-focused names
- Detailed docstrings on every Python function, method, fixture, helper, and test touched by the change; missing function documentation is not acceptable in
tests/integration/
- Proper skip guards for missing keys, App Gateway absence, or disabled features
- Coverage built on the shared Python runtime instead of ad hoc curl/bash logic
- No hardcoded URLs, keys, or model names outside the centralized config/runtime layer
External Documentation
- Use External Docs Research as the single source of truth for external documentation workflow and fallback approval requirements.
Documentation Sync
- If the change adds, removes, renames, or materially reorganizes tracked files or directories, update the root
README.md Folder Structure section in the same change. Do not add gitignored or local-only artifacts to that tree.
- Review the documentation sync matrix in ../../copilot-instructions.md and update any area-specific README or docs pages it calls out for the touched subtree.
Scope
- Test framework: pytest
- Project root:
tests/integration/
- Runtime modules:
src/ai_hub_integration/config.py, client.py, runner.py
- Live suites:
tests/test_*.py
- Unit tests:
tests/unit/
- CLI entrypoints:
run-tests.py, run-tests.sh
- Optional evaluation CLI:
run-evaluation.py
Suite Inventory
| File | Focus |
|---|
tests/test_chat_completions.py | OpenAI deployment-route coverage centered on ai-hub-admin with NR-DAP smoke checks |
tests/test_v1_chat_completions.py | OpenAI /openai/v1 format, streaming SSE, Bearer auth, and rate-limit regression checks |
tests/test_document_intelligence.py | JSON and async Document Intelligence coverage centered on ai-hub-admin |
tests/test_document_intelligence_binary.py | WAF/App Gateway allow-rules for binary and multipart Document Intelligence uploads |
tests/test_app_gateway.py | App Gateway TLS, routing, auth normalization, and the retained cross-key verification |
tests/test_apim_key_rotation.py | /internal/apim-keys endpoint, rotation metadata, and Key Vault fallback coverage |
tests/test_tenant_info.py | Tenant info endpoint, model metadata, and service feature flags for ai-hub-admin |
tests/test_mistral.py | Mistral chat and OCR routing via APIM |
tests/test_ai_evaluation.py | Optional Azure AI Evaluation SDK scoring |
Execution Model
run-tests.py is the authoritative runner. It provides:
- Suite alias mapping from legacy names (
chat-completions, apim-key-rotation, etc.)
- Marker-based grouping:
all, direct, proxy
- Optional inclusion of
ai_eval suites
The shell wrapper run-tests.sh exists for convenience and backwards-compatible local usage, but it delegates to run-tests.py.
Running Tests
cd tests/integration
uv sync --group dev
uv run pytest tests/unit -q
uv run python ./run-tests.py --env test --group all
uv run python ./run-tests.py --env test --group direct
HTTP_PROXY=http://127.0.0.1:8118 \
HTTPS_PROXY=http://127.0.0.1:8118 \
uv run python ./run-tests.py --env test --group proxy
Skip Patterns
| Pattern | When to use |
|---|
require_key(config, tenant) | Test requires a tenant subscription key |
require_appgw(config) | Test requires App Gateway to be deployed |
integration_config.is_apim_key_rotation_enabled() | Key-rotation endpoint can be disabled per environment |
pytest.skip(...) | Inline skip for unavailable infrastructure or optional evaluation config |
CI Model
.github/workflows/.integration-tests-using-secure-tunnel.yml splits execution into:
- Direct:
uv run python ./run-tests.py --group direct
- Optional AI Evaluation:
uv run python ./run-evaluation.py when judge-model secrets and vars exist
- Proxy:
uv run python ./run-tests.py --group proxy with HTTP_PROXY/HTTPS_PROXY set to the Bastion SOCKS tunnel (fronted by privoxy)
Validation Gates (Required)
uv sync --group dev succeedsuv run ruff check . is cleanuv run ruff format --check . is cleanuv run pytest tests/unit -q passes- Live tests keep retry-safe behavior for 429/503-prone endpoints
- Async Document Intelligence flows still poll through
wait_for_operation
- Every Python function and method in the touched integration files has a meaningful docstring before handoff
Detailed References
For helper signatures, config loading rules, retry behavior, secure-tunnel expectations, and failure playbooks, see references/REFERENCE.md.
