scaffold-connector

# OpenMetadata Connector Building Skill ## When to Activate When a user asks to build, create, add, or scaffold a new connector, source, or integration for OpenMetadata. ## Core Insight **One JSON Schema definition cascades through 6 layers**: Python Pydantic models, Java models, UI forms (RJSF auto-render), API validation, test fixtures, and documentation. Define the schema once — everything else is generated or guided. ## Workflow: 7 Phases ### Phase 0: ENVIRONMENT — Set Up Python Dev Environment Before any `make` or `python` commands, set up the environment from the repo root: ```bash python3.11 -m venv env source env/bin/activate make install_dev generate ``` Always activate before running commands: `source env/bin/activate` ### Phase 1: SCAFFOLD — Generate Boilerplate Run the scaffold CLI to collect inputs and generate files: ```bash source env/bin/activate metadata scaffold-connector ``` Interactive mode collects: connector name, service type, connection type, auth types, capabilities, docs URL, SDK package, API endpoints, implementation notes, Docker image, container port. Non-interactive mode: ```bash metadata scaffold-connector \ --name my_db \ --service-type database \ --connection-type sqlalchemy \ --scheme "mydb+pymydb" \ --auth-types basic \ --capabilities metadata lineage usage profiler \ --docs-url "https://docs.example.com/api" \ --sdk-package "mydb-sdk" \ --docker-image "mydb/mydb:latest" \ --docker-port 5432 ``` **Output**: JSON Schema + test connection JSON + Python files + `CONNECTOR_CONTEXT.md` as an AI working document. SQLAlchemy database connectors get concrete code templates; all others get skeleton files with pointers to reference connectors. **CONNECTOR_CONTEXT.md handling**: The scaffold generates `CONNECTOR_CONTEXT.md` in the connector directory as a working document for any AI tool (Claude Code, Cursor, Codex, Copilot, Windsurf). It is **gitignored** — it stays local and is never committed to the repo. No cleanup needed. ### Phase 2: CLASSIFY — Understand the Source The scaffold classifies along 3 dimensions. Verify the choices: **Dimension 1 — Service Type** (determines directory + base class): | Service Type | Base Class | Reference | |---|---|---| | `database` | `CommonDbSourceService` | `mysql/` | | `dashboard` | `DashboardServiceSource` | `metabase/` | | `pipeline` | `PipelineServiceSource` | `airflow/` | | `messaging` | `MessagingServiceSource` | `kafka/` | | `mlmodel` | `MlModelServiceSource` | `mlflow/` | | `storage` | `StorageServiceSource` | `s3/` | | `search` | `SearchServiceSource` | `elasticsearch/` | | `api` | `ApiServiceSource` | `rest/` | **Dimension 2 — Connection Type** (database only): - `sqlalchemy` → `BaseConnection[Config, Engine]` + SQLAlchemy dialect - `rest_api` → `get_connection()` + custom REST client (ref: `salesforce/`) - `sdk_client` → `get_connection()` + vendor SDK wrapper **Dimension 3 — Capabilities** (determines extra files): `metadata` (always), `lineage`, `usage`, `profiler`, `stored_procedures`, `data_diff` Read the source-type-specific standard at `${CLAUDE_SKILL_DIR}/standards/source_types/{service_type}.md` for detailed patterns. ### Phase 3: RESEARCH — API/SDK Discovery Read the `CONNECTOR_CONTEXT.md` generated by the scaffold. Then research the source's API/SDK. **If you can dispatch sub-agents** (Claude Code): Launch a `connector-researcher` agent: ``` Agent: openmetadata-skills:connector-researcher Prompt: "Research {source_name} for an OpenMetadata {service_type} connector. Find: API docs, auth methods, key endpoints, pagination, rate limits, SDK packages." ``` **If you cannot dispatch sub-agents**: Perform the research yourself using WebSearch and WebFetch. ### Phase 4: IMPLEMENT — Fill in the TODO Items The scaffold generates files with `# TODO` markers. Read the relevant standards before implementing: - `${CLAUDE_SKILL_DIR}/standards/connection.md` — Connection patterns - `${CLAUDE_SKILL_DIR}/standards/patterns.md` — Error handling, pagination, auth - `${CLAUDE_SKILL_DIR}/standards/performance.md` — Pagination, lookup optimization, anti-patterns - `${CLAUDE_SKILL_DIR}/standards/memory.md` — Memory management, streaming, OOM prevention - `${CLAUDE_SKILL_DIR}/standards/source_types/{service_type}.md` — Service-specific patterns **SQLAlchemy database**: Templates are mostly complete. Customize `_get_client()` if needed. **Non-SQLAlchemy**: Study the reference connector, then implement each skeleton file. **Critical for JSON Schema**: - Make auth fields (`username`, `password`, `token`) **required** when the service needs authentication by default. If omitting a field means an opaque 401 at runtime, make it required so the UI validates upfront. - Include SSL/TLS config (`verifySSL` + `sslConfig` `$ref`) for any connector that communicates over HTTPS — enterprise deployments use internal CAs. - **SSL must be wired end-to-end**: schema → `connection.py` (resolve with `get_verify_ssl_fn`) → `client.py` (`session.verify = verify_ssl`). Missing wiring triggers SonarQube Security Review failure. - See `${CLAUDE_SKILL_DIR}/standards/schema.md` for the `$ref` patterns and required fields guidance. **Critical for Pydantic API models (models.py)**: - Always set `model_config = ConfigDict(populate_by_name=True)` when using `Field(alias=...)` — without this, constructing instances with Python attribute names raises `ValidationError`. - See `${CLAUDE_SKILL_DIR}/standards/code_style.md` for the full pattern. **Critical for non-database connectors (client.py)**: - Every list endpoint MUST implement pagination if the API supports it. Check the API docs. - Missing pagination causes silent data loss — only the first page is ingested. - Build dicts for repeated lookups (e.g., folder path → folder name) instead of iterating lists. - See `${CLAUDE_SKILL_DIR}/standards/performance.md` for correct patterns and anti-patterns. **Critical for storage connectors and any connector that reads files**: - Never `.read()` entire files without a size check — causes OOM on production instances. - Use framework streaming readers (`metadata/readers/dataframe/`) for data files. - `del` large objects after processing and call `gc.collect()`. - See `${CLAUDE_SKILL_DIR}/standards/memory.md` for correct patterns. **Critical for lineage**: - Never use wildcard `table_name="*"` in search queries — this links every table in a database to each entity, producing incorrect lineage. - If the source doesn't provide table-level info, skip lineage and document the limitation. - See `${CLAUDE_SKILL_DIR}/standards/lineage.md` for correct patterns. ### Phase 5: REGISTER — Integration Points Read `${CLAUDE_SKILL_DIR}/standards/registration.md` for detailed instructions. Summary: | Step | File | Change | |------|------|--------| | 1 | `openmetadata-spec/.../entity/services/{serviceType}Service.json` | Add to type enum + connection oneOf | | 2 | `ingestion/setup.py` | Add connector pip extras block | | 3 | `ingestion/src/metadata/examples/workflows/{name}.yaml` | Runnable CLI workflow example (`metadata ingest -c`) | | 4 | `openmetadata-ui/.../utils/{ServiceType}ServiceUtils.tsx` | Import schema + add switch case | | 5 | `openmetadata-ui/.../assets/img/service-icon-{name}.png` | Add service logo asset | | 6 | `openmetadata-ui/.../utils/ServiceIconUtils.ts` | Import asset + register in `SERVICE_ICON_LOADERS` | | 7 | `openmetadata-ui/.../public/locales/en-US/{ServiceType}/{Name}.md` | Field-level docs rendered in the Add Service form | | 8 | `openmetadata-ui/.../constants/ServiceType.constant.ts` | Append to `BETA_SERVICES` — new connectors always ship as Beta | > i18n locale files do **not** need edits — display names come from the generated service type enum. `Services.constant.ts` is deprecated (re-export shim only); register icons in `ServiceIconUtils.ts`. ### Phase 6: GENERATE & FORMAT — Run Code Generation and Formatting This step is **mandatory** — always run it before committing. Ensure the Python environment is set up: ```bash # Ensure environment is active and tools are installed source env/bin/activate pip install -e ".[dev]" 2>/dev/null || make install_dev # Generate models from schemas make generate # Python Pydantic models mvn clean install -pl openmetadata-spec # Java models cd openmetadata-ui/src/main/resources/ui && yarn parse-schema # UI schemas # Format ALL code (mandatory before commit) cd /path/to/repo/root make py_format # ruff lint-fix + format mvn spotless:apply # Format Java ``` **If `make py_format` fails**: The most common cause is missing dev dependencies. Run `make install_dev` first, then retry. **Never skip formatting** — unformatted code will fail CI. ### Phase 7: VALIDATE — Run Static Analysis and Checklist Run the static analyzer as a self-check before submitting: ```bash python skills/connector-review/scripts/analyze_connector.py {service_type} {name} ``` Fix any issues it reports. Then verify the full checklist: ``` [ ] JSON Schema: validates, $ref resolves, supports* flags correct [ ] JSON Schema: auth fields required when service mandates authentication [ ] JSON Schema: SSL/TLS config included for HTTPS connectors [ ] Code gen: make generate + mvn install + yarn parse-schema succeed [ ] Connection: creates client, test_connection passes all steps [ ] Source: create() validates config type, ServiceSpec is discoverable [ ] Pydantic models: populate_by_name=True on all aliased models [ ] Client: all list endpoints paginate (check API docs for pagination support) [ ] Client: dict lookups in prepare(), not list iteration per entity [ ] Lineage: no wildcard table_name="*" — skip if no table-level info available [ ] Tests: unit + connection integration + metadata integration pass (no empty stubs) [ ] Registration: service schema, setup.py, {ServiceType}ServiceUtils.tsx all updated [ ] Registration: CLI workflow example in ingestion/src/metadata/examples/workflows/ [ ] Registration: service icon asset present + registered in ServiceIconUtils.ts [ ] Registration: docs markdown in public/locales/en-US/{ServiceType}/{Name}.md [ ] Registration: service type appended to BETA_SERVICES in ServiceType.constant.ts [ ] Formatting: make py_format + mvn spotless:apply pass with no changes [ ] Cleanup: CONNECTOR_CONTEXT.md is gitignored (verify it's not staged) [ ] Cleanup: no leftover TODO scaffolding comments ``` ### Phase 8: TEST LOCALLY — Deploy and Test in the UI Build everything and bring up a full local OpenMetadata stack with Docker: **Full build** (first time or after Java/UI changes): ```bash ./docker/run_local_docker.sh -m ui -d mysql -s false -i true -r true ``` **Fast rebuild** (ingestion-only changes, ~2-3 minutes): ```bash ./docker/run_local_docker.sh -m ui -d mysql -s true -i true -r false ``` Once services are up (~3-5 minutes): 1. Open **http://localhost:8585** 2. Go to **Settings → Services → {Your Service Type}** 3. Click **Add New Service** and select your connector 4. Configure connection details and click **Test Connection** 5. If test passes, run metadata ingestion to verify entities are created Other service URLs: - Airflow: http://localhost:8080 (admin / admin) - Elasticsearch: http://localhost:9200 **Tear down**: `cd docker/development && docker compose down -v` **Troubleshooting**: - Connector not in dropdown → check service schema registration, rebuild without `-s true` - Test connection fails → check `test_fn` keys match test connection JSON step names - Container logs: `docker compose -f docker/development/docker-compose.yml logs ingestion` ### Phase 9: CREATE PR — Submit with Quality Summary When creating a PR for the connector, include the review summary in the PR description so reviewers see the quality assessment upfront: ```bash # Run the static analyzer analysis=$(python skills/connector-review/scripts/analyze_connector.py {service_type} {name} --json) # Create PR with quality summary in description gh pr create --title "feat(ingestion): Add {Name} {service_type} connector" --body "$(cat <<'EOF' ## Summary - New {service_type} connector for {Name} - Capabilities: {list capabilities} ## Test plan - [ ] Unit tests pass (`pytest ingestion/tests/unit/topology/{service_type}/test_{name}.py`) - [ ] Integration tests pass - [ ] Local Docker test: connector appears in UI, test connection passes ## Connector Quality Review **Verdict**: {VERDICT} | **Score**: {SCORE}/10 | Category | Score | |----------|-------| | Schema & Registration | X/10 | | Connection & Auth | X/10 | | Source, Topology & Performance | X/10 | | Test Quality | X/10 | | Code Quality & Style | X/10 | **Blockers**: 0 | **Warnings**: {count} | **Suggestions**: {count} <details> <summary>Static analysis output</summary> {paste analyze_connector.py output here} </details> 🤖 Generated with [Claude Code](https://claude.com/claude-code) EOF )" ``` The quality summary gives maintainers confidence about the connector's state without needing to review every file manually. ## Standards Reference All standards are in `${CLAUDE_SKILL_DIR}/standards/`: | Standard | Content | |----------|---------| | `main.md` | Architecture overview, connector anatomy, service types | | `patterns.md` | Error handling, logging, pagination, auth, filters | | `testing.md` | Unit test patterns, integration tests, pytest style | | `code_style.md` | Python style, JSON Schema conventions, naming | | `schema.md` | Connection schema patterns, $ref usage, test connection JSON | | `connection.md` | BaseConnection vs function patterns, SSL, client wrapper | | `service_spec.md` | DefaultDatabaseSpec vs BaseSpec | | `registration.md` | Service enum, UI utils, i18n | | `performance.md` | Pagination, batching, rate limiting | | `memory.md` | Memory management, streaming, OOM prevention | | `lineage.md` | Lineage extraction methods, dialect mapping, query logs |