connector-doc-create

# Connector Documentation Creation Skill ## When to Activate When a user asks to create, generate, scaffold, or add documentation for a **new** connector — one that exists in source code but doesn't yet have docs pages. ## Arguments - **connector-name** (required): Slug name of the connector (e.g., `redshift`, `dynamodb`, `bigquery`, `airflow`, `looker`, `kafka`). Must match the directory name in `${SOURCE_ROOT}/${service_type}/`. - **--display-name** (optional): Human-readable name shown in docs (e.g., `"Amazon Redshift"`, `"Google BigQuery"`). Defaults to title-cased connector-name. - **--service-type** (optional): One of `database`, `pipeline`, `dashboard`, `messaging`, `storage`, `search`, `ml-model`. Note: use `ml-model` (with hyphen) — this matches the actual directory name. Other types (`api`, `drive`, `metadata`) exist in the repo but are rare; treat them like `pipeline` if encountered. Default: auto-detect from connector name and schema. - **--stage** (optional): `PROD` or `BETA`. Default: `BETA` for new connectors. - **--version** (optional): Which version(s) to create docs for. Default: `all` (creates in v1.13.x, v2.0.x). - **--icon** (optional): Path to icon file under `/public/images/connectors/`. If not given, uses `/public/images/connectors/{connector-name}.svg` as a placeholder. - **--dry-run** (optional): Only show the plan and generated content — don't write any files. ## Directory References ``` DOCS_ROOT = . # The docs-om repo (current working directory) OM_ROOT = ../OpenMetadata # Sibling directory to docs-om SCHEMA_ROOT = ${OM_ROOT}/openmetadata-spec/src/main/resources/json/schema/entity/services SOURCE_ROOT = ${OM_ROOT}/ingestion/src/metadata/ingestion/source ``` --- ## Creation Process ### Phase 1: Read Ground Truth from Source Code #### Step 1.1: Read the JSON Schema ``` Schema file: ${SCHEMA_ROOT}/connections/${service_type}/${connectorName}Connection.json ``` Extract: 1. **All properties** — field names, types, descriptions, required fields, defaults 2. **`supports*` boolean flags** — determines available/unavailable features 3. **Filter pattern fields** — `schemaFilterPattern`, `tableFilterPattern`, `storedProcedureFilterPattern`, etc. 4. **`sampleDataStorageConfig`** — presence indicates Sample Data support 5. **Authentication types** — `authType` property's `oneOf`/`anyOf` references: - `basicAuth.json` → **Basic Auth** - `iamAuthConfig.json` → **IAM Auth** - `awsCredentials.json` → **AWS Credentials** - `gcpCredentials.json` → **GCP Credentials** - `azureCredentials.json` → **Azure Credentials** 6. **SSL fields** — `sslMode`, `sslConfig`, `verifySSL` 7. **Required fields** — the `required` array Also check the parent service schema to confirm registration: ``` ${SCHEMA_ROOT}/${serviceType}Service.json ``` #### Step 1.2: Read the Source Code Read these files: ``` ${SOURCE_ROOT}/${service_type}/${connector_name}/metadata.py — Source class, capabilities ${SOURCE_ROOT}/${service_type}/${connector_name}/connection.py — Connection logic, test steps, permissions ${SOURCE_ROOT}/${service_type}/${connector_name}/service_spec.py — Registered source classes ``` Also check if these exist: ``` ${SOURCE_ROOT}/${service_type}/${connector_name}/queries.py — SQL queries (reveals permission requirements) ${SOURCE_ROOT}/${service_type}/${connector_name}/client.py — REST API client ``` Extract: 1. **Test connection steps** — `test_fn` dict keys in `test_connection()` → permission names 2. **Service spec classes** — lineage_source_class, usage_source_class, profiler_class (confirm feature support) 3. **Mixin/base classes** — what the source extends (e.g., `LifeCycleQueryMixin`, `MultiDBSource`) 4. **Owner/Tag extraction** — search for `yield_tag`, `yield_table_tag`, `get_tag_labels`, owner-related methods 5. **Permission hints** — SQL grants, IAM actions, API scopes in docstrings or comments 6. **Python package name** — `pip install "openmetadata-ingestion[{package}]"` (check pyproject.toml or setup.py if unsure) #### Step 1.3: Build the Feature Truth Table Using schema + code, build the definitive feature map: **For Database Connectors:** | Signal | Feature | |---|---| | `supportsMetadataExtraction: true` (default) | "Metadata" → available | | `supportsUsageExtraction: true` | "Query Usage" → available | | `supportsLineageExtraction: true` | "Lineage" or "View Lineage" → available | | `supportsViewLineageExtraction: true` | "Column-level Lineage" → available | | `supportsProfiler: true` | "Data Profiler" → available | | `supportsProfiler: true` (implicit) | "Auto-Classification" → available | | `supportsDBTExtraction: true` | "dbt" → available | | `supportsDataDiff: true` | "Data Quality" → available | | `storedProcedureFilterPattern` present | "Stored Procedures" → available | | `sampleDataStorageConfig` present | "Sample Data" → available | | Owner extraction code found | "Owners" → available | | Tag extraction code found | "Tags" → available | **For Pipeline Connectors:** | Signal | Feature | |---|---| | Always | "Pipelines" → available | | Status extraction code found | "Pipeline Status" → available | | lineage_source_class in service_spec | "Lineage" → available | | Owner extraction code found | "Owners" → available | | Tag extraction code found | "Tags" → available | **For Dashboard Connectors:** | Signal | Feature | |---|---| | Always | "Dashboards", "Charts" → available | | Datamodel extraction code found | "Datamodels" → available | | Project extraction code found | "Projects" → available | | lineage_source_class in service_spec | "Lineage" → available | | Column lineage code found | "Column Lineage" → available | | Owner extraction code found | "Owners" → available | | usage_source_class in service_spec | "Usage" → available | | Tag extraction code found | "Tags" → available | **For Messaging Connectors:** | Signal | Feature | |---|---| | Always | "Topics" → available | | `sampleDataStorageConfig` present | "Sample Data" → available | **For Storage Connectors:** | Signal | Feature | |---|---| | Always | "Metadata" → available | | Structured container code found | "Structured Containers" → available | | Unstructured container code found | "Unstructured Containers" → available | **For Search Connectors:** | Signal | Feature | |---|---| | Always | "Search Indexes" → available | | Sample data code found | "Sample Data" → available | **For ML Model Connectors:** | Signal | Feature | |---|---| | Always | "ML Features" → available | | Hyperparameter code found | "Hyperparameters" → available | | ML store code found | "ML Store" → available | #### Step 1.4: Build the Permissions List From the extracted code, derive the permission requirements: - **Database connectors**: Map SQL queries to required GRANT statements. Common pattern: - Reading system views/tables → `SELECT` on those views - `information_schema` reads → `USAGE` on schema - Query history access → specific grants (e.g., `pg_read_all_stats`) - **AWS connectors**: Map API calls to IAM actions (e.g., `dynamodb:ListTables`, `s3:GetObject`) - **GCP connectors**: Map API calls to GCP roles/permissions - **REST API connectors**: Map endpoints to required API scopes/roles Group permissions by capability: 1. Metadata Ingestion 2. Profiler & Data Quality (if supported) 3. Usage & Lineage (if supported) #### Step 1.5: Extract YAML Configuration Fields From the JSON schema `properties`, build a YAML config template: 1. Start with `required` fields — these MUST appear, no comments 2. Add important optional fields with commented-out examples 3. Use placeholder values: `<username>`, `<password>`, `<hostname>`, `<database>` 4. For auth type fields (`oneOf`/`anyOf`): show the most common auth type uncommented, others as commented blocks 5. For filter patterns: show commented-out example with `includes`/`excludes` 6. For SSL: show commented-out block Build ContentSection entries for each field shown in YAML: - Match the YAML key name exactly - Use the schema `description` as the base, expand if needed - For complex fields (auth, SSL), add links to relevant docs pages --- ### Phase 2: Check for Existing Files Before creating anything, check: 1. Does `${DOCS_ROOT}/{version}/connectors/${service_type}/${connector_name}.mdx` already exist? 2. Does `${DOCS_ROOT}/{version}/connectors/${service_type}/${connector_name}/yaml.mdx` already exist? 3. Is the connector already registered in `docs.json`? If files exist, **stop and warn the user** — use the `connector-doc-review` skill instead to update existing docs. --- ### Phase 3: Generate Documentation Files For each target version (v1.13.x, v2.0.x): #### Step 3.1: Generate the Main Page (`{connector_name}.mdx`) **File path:** `${DOCS_ROOT}/{version}/connectors/${service_type}/${connector_name}.mdx` **Template — varies by service type. Use the correct imports and structure for the connector's service type:** **Shared imports (all service types):** ```mdx import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx' import TestConnection from '/snippets/connectors/test-connection.mdx' import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx' import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx' ``` **Service-type-specific imports for main page:** | Service Type | ConfigureIngestion | Extra imports | |---|---|---| | `database` | `/snippets/connectors/database/configure-ingestion.mdx` | `AdvancedConfiguration` from `/snippets/connectors/database/advanced-configuration.mdx`; `Related` from `/snippets/{version}/connectors/database/related.mdx` | | `pipeline` | `/snippets/connectors/pipeline/configure-ingestion.mdx` | — | | `dashboard` | `/snippets/connectors/dashboard/configure-ingestion.mdx` | — | | `messaging` | `/snippets/connectors/messaging/configure-ingestion.mdx` | — | | `storage` | `/snippets/connectors/storage/configure-ingestion.mdx` | `Manifest` from `/snippets/connectors/storage/manifest.mdx`; **do NOT use MetadataIngestionUi** | | `search` | `/snippets/connectors/search/configure-ingestion.mdx` | — | | `ml-model` | `/snippets/connectors/ml-model/configure-ingestion.mdx` | — | **Steps block — varies by service type:** - **Database only**: include `<AdvancedConfiguration />` between Connection Details Step and `<TestConnection />` - **All types**: include `<TestConnection />`, `<ConfigureIngestion />`, `<IngestionScheduleAndDeploy />` - **Database only**: include `<Related />` at bottom of page - **Storage only**: include `<Manifest />` in requirements section (OpenMetadata manifest file is required) **MetadataIngestionUi**: Used by database, pipeline, dashboard, search, ml-model. **NOT used by storage** (storage uses manual step descriptions instead). **Full template:** ```mdx --- title: "{DisplayName} Connector | OpenMetadata {ServiceType} Integration" description: "Connect {DisplayName} to OpenMetadata with our comprehensive {service_type} connector guide. Step-by-step setup, configuration examples, and metadata extraction tips." sidebarTitle: Overview --- {service-type-specific imports — see table above} <ConnectorDetailsHeader icon='/public/images/connectors/{connector_name}.svg' name="{DisplayName}" stage="{PROD|BETA}" availableFeatures={[{availableFeatures}]} unavailableFeatures={[{unavailableFeatures}]} /> In this section, we provide guides and references to use the {DisplayName} connector. {IF multiple auth types: include <Info> callout listing them} Configure and schedule {DisplayName} metadata workflows from the OpenMetadata UI: - [Requirements](#requirements) - [Metadata Ingestion](#metadata-ingestion) {IF database AND Query Usage supported: - [Query Usage](/{version}/connectors/ingestion/workflows/usage)} {IF database AND Profiler supported: - [Data Profiler](/{version}/how-to-guides/data-quality-observability/profiler/profiler-workflow)} {IF database AND Data Quality supported: - [Data Quality](/{version}/how-to-guides/data-quality-observability/quality)}