add-connector

# Add Connector Skill ## Choose the connector runtime first For **Sim Search**, use the live provider workflow in [the federated Search developer guide](../../../apps/sim/lib/sim-search/live/README.md#adding-a-live-search-connector). Its browser-safe provider catalog owns provider IDs, API origins, credential aliases, and account modes; its typed runtime registry requires both search and read handlers. `ConnectorMeta` remains the owner of logos and setup fields. Member mode has no admin resource filters. Service mode requires independent live source verification; resource pickers use shared selectors with canonical manual-input pairs, and plain inputs are fine where no picker applies. Do not implement a Search source by adding a crawler, embeddings, or a scheduled ACL build. The ingestion instructions below apply to **ordinary knowledge-base connectors**. Sim Search always uses the live backend, so setting `search: true` in metadata does not implement federated search; a provider that serves both needs the live handlers and this KB ingestion path. You are an expert at adding knowledge base connectors to Sim. A connector syncs documents from an external source (Confluence, Google Drive, Notion, etc.) into a knowledge base. ## Your Task When the user asks you to create a connector: 1. Use Context7 or WebFetch to read the service's API documentation 2. Determine the auth mode: **OAuth** (if Sim already has an OAuth provider for the service) or **API key** (if the service uses API key / Bearer token auth) 3. Create the connector directory: a client-safe `meta.ts` (declarative metadata) plus the runtime module that spreads it 4. Register it in BOTH the server registry and the client-safe meta registry ## Hard Rule: No Guessed Response Or Document Schemas If the service docs do not clearly show the document list response, document fetch response, pagination shape, or metadata fields, you MUST tell the user instead of guessing. - Do NOT invent document fields - Do NOT guess pagination cursors or next-page fields - Do NOT infer metadata/tag mappings from unrelated endpoints - Do NOT fabricate `ExternalDocument` content structure from partial docs If the source schema is unknown, do one of these instead: 1. Ask the user for sample API responses 2. Ask the user for test credentials so you can verify live payloads 3. Implement only the documented parts of the connector 4. Leave the connector incomplete and explicitly say which fields remain unknown ## Directory Structure Each connector is split into a client-safe metadata file and a server-only runtime file. This mirrors the `XBlockMeta` / `BLOCK_META_REGISTRY` split in `apps/sim/blocks` — client components (the knowledge UI) only need the metadata (icon, name, auth, config fields), so the runtime functions (which pull server-only helpers like `input-validation.server` → `undici` → `node:net`) must stay out of the client bundle. Create files in `apps/sim/connectors/{service}/`: ``` connectors/{service}/ ├── index.ts # Barrel export (re-exports the runtime connector) ├── meta.ts # ConnectorMeta — client-safe declarative metadata └── {service}.ts # ConnectorConfig — spreads the meta + adds runtime functions ``` - `meta.ts` exports `{service}ConnectorMeta: ConnectorMeta`. It imports ONLY the icon from `@/components/icons`, `import type { ConnectorMeta } from '@/connectors/types'`, and any pure-data constants. It must NEVER import server/runtime code. - `{service}.ts` exports `{service}Connector: ConnectorConfig`. It imports the meta via `import { {service}ConnectorMeta } from '@/connectors/{service}/meta'`, spreads it as the first property, and holds the runtime functions (which may import server-only helpers like `@/lib/knowledge/documents/utils`). ## Authentication Connectors use a discriminated union for auth config (`ConnectorAuthConfig` in `connectors/types.ts`): See `ConnectorAuthConfig` in `apps/sim/connectors/types.ts`: `oauth` takes `provider`, `requiredScopes`, and optional service-account/admin scopes; `apiKey` takes `label`, `placeholder`, and `optional`. ### OAuth mode For services with existing OAuth providers in `apps/sim/lib/oauth/types.ts`. The `provider` must match an `OAuthService`. The modal shows a credential picker and handles token refresh automatically. ### API key mode For services that use API key / Bearer token auth. The modal shows a password input with the configured `label` and `placeholder`. The API key is encrypted at rest using AES-256-GCM and stored in a dedicated `encryptedApiKey` column on the connector record. The sync engine decrypts it automatically — connectors receive the raw access token in `listDocuments`, `getDocument`, and `validateConfig`. ## Connector Structure (meta.ts + runtime) The declarative metadata lives in `meta.ts` (`ConnectorMeta`). The runtime functions live in `{service}.ts` (`ConnectorConfig`), which spreads the meta as its first property. ### `meta.ts` — client-safe metadata ```typescript import { {Service}Icon } from '@/components/icons' import type { ConnectorMeta } from '@/connectors/types' export const {service}ConnectorMeta: ConnectorMeta = { id: '{service}', name: '{Service}', description: 'Sync documents from {Service} into your knowledge base', version: '1.0.0', icon: {Service}Icon, auth: { mode: 'oauth', provider: '{service}', // Must match OAuthService in lib/oauth/types.ts requiredScopes: ['read:...'], }, configFields: [ // Rendered dynamically by the add-connector modal UI // Supports 'short-input', 'dropdown', and 'selector' types — see ConfigField Types below ], // Optional: tag definitions are metadata too — declare them here // tagDefinitions: [ ... ], } ``` Keep `meta.ts` free of any server/runtime import. Only the icon, the `ConnectorMeta` type, and pure-data constants belong here. ### `{service}.ts` — runtime (OAuth example) ```typescript import { createLogger } from '@sim/logger' import { fetchWithRetry } from '@/lib/knowledge/documents/secure-fetch.server' import { {service}ConnectorMeta } from '@/connectors/{service}/meta' import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types' const logger = createLogger('{Service}Connector') export const {service}Connector: ConnectorConfig = { ...{service}ConnectorMeta, listDocuments: async (accessToken, sourceConfig, cursor) => { // Return metadata stubs with contentDeferred: true (if per-doc content fetch needed) // Or full documents with content (if list API returns content inline) // Return { documents: ExternalDocument[], nextCursor?, hasMore } }, getDocument: async (accessToken, sourceConfig, externalId) => { // Fetch full content for a single document // Return ExternalDocument with contentDeferred: false, or null }, validateConfig: async (accessToken, sourceConfig) => { // Return { valid: true } or { valid: false, error: 'message' } }, // Optional: map source metadata to semantic tag keys (translated to slots by sync engine) mapTags: (metadata) => { // Return Record<string, unknown> with keys matching tagDefinitions[].id }, } ``` Only map fields in `listDocuments`, `getDocument`, `validateConfig`, and `mapTags` when the source payload shape is documented or live-verified. If not, tell the user and stop rather than guessing. ### API key connector example The split is identical — `auth` lives in `meta.ts`, runtime functions in `{service}.ts`. ```typescript // meta.ts export const {service}ConnectorMeta: ConnectorMeta = { id: '{service}', name: '{Service}', description: 'Sync documents from {Service} into your knowledge base', version: '1.0.0', icon: {Service}Icon, auth: { mode: 'apiKey', label: 'API Key', // Shown above the input field placeholder: 'Enter your {Service} API key', // Input placeholder }, configFields: [ /* ... */ ], } // {service}.ts export const {service}Connector: ConnectorConfig = { ...{service}ConnectorMeta, listDocuments: async (accessToken, sourceConfig, cursor) => { /* ... */ }, getDocument: async (accessToken, sourceConfig, externalId) => { /* ... */ }, validateConfig: async (accessToken, sourceConfig) => { /* ... */ }, } ``` ## ConfigField Types The add-connector modal renders these automatically — no custom UI needed. Three field types are supported: `short-input`, `dropdown`, and `selector`. ```typescript // Text input { id: 'domain', title: 'Domain', type: 'short-input', placeholder: 'yoursite.example.com', required: true, } // Dropdown (static options) { id: 'contentType', title: 'Content Type', type: 'dropdown', required: false, options: [ { label: 'Pages only', id: 'page' }, { label: 'Blog posts only', id: 'blogpost' }, { label: 'All content', id: 'all' }, ], } ``` ## Dynamic Selectors (Canonical Pairs) Use `type: 'selector'` for a key declared in the browser-safe selector manifest at `apps/sim/lib/selectors/manifest.ts`. Remote selectors execute through the authorized `selectors.execute` server operation and a server attachment; connectors never call providers or resolve credentials in the browser. Apply the `add-selector` skill when the key does not exist. Selectors are paired with a manual fallback input using the **canonical pair** pattern — a `selector` field (basic mode) and a `short-input` field (advanced mode) linked by `canonicalParamId`. The user sees a toggle button (ArrowLeftRight) to switch between the selector dropdown and manual text input. On submit, the modal resolves each canonical pair to the active mode's value, keyed by `canonicalParamId`. ### Rules 1. **Every selector field MUST have a canonical pair** — a corresponding `short-input` (or `dropdown`) field with the same `canonicalParamId` and `mode: 'advanced'`. 2. **`required` must be set identically on both fields** in a pair. If the selector is required, the manual input must also be required. 3. **`canonicalParamId` must match the key the connector expects in `sourceConfig`** (e.g. `baseId`, `channel`, `teamId`). The advanced field's `id` should typically match `canonicalParamId` (connector config fields differ from block subBlocks here; the block rule that `canonicalParamId` must not equal the id of a subblock without a `canonicalParamId` does not apply). 4. **`dependsOn` references the selector field's `id`**, not the `canonicalParamId`. The modal propagates dependency clearing across canonical siblings automatically — changing either field in a parent pair clears dependent children. ### Selector canonical pair example (Airtable base → table cascade) ```typescript configFields: [ // Base: selector (basic) + manual (advanced) { id: 'baseSelector', title: 'Base', type: 'selector', selectorKey: 'airtable.bases', // Must exist in lib/selectors/manifest.ts canonicalParamId: 'baseId', mode: 'basic', placeholder: 'Select a base', required: true, }, { id: 'baseId', title: 'Base ID', type: 'short-input', canonicalParamId: 'baseId', mode: 'advanced', placeholder: 'e.g. appXXXXXXXXXXXXXX', required: true, }, // Table: selector depends on base (basic) + manual (advanced) { id: 'tableSelector', title: 'Table', type: 'selector', selectorKey: 'airtable.tables', canonicalParamId: 'tableIdOrName', mode: 'basic', dependsOn: ['baseSelector'], // References the selector field ID placeholder: 'Select a table', required: true, }, { id: 'tableIdOrName', title: 'Table Name or ID', type: 'short-input', canonicalParamId: 'tableIdOrName', mode: 'advanced', placeholder: 'e.g. Tasks', required: true, }, // Non-selector fields stay as-is { id: 'maxRecords', title: 'Max Records', type: 'short-input', ... }, ] ``` ### Selector with domain dependency (Jira/Confluence pattern) When a selector depends on a plain `short-input` field (no canonical pair), `dependsOn` references that field's `id` directly. Exact references such as `{{JIRA_DOMAIN}}` remain unresolved in the browser and are resolved only after workspace authorization on the server. ```typescript configFields: [ { id: 'domain', title: 'Jira Domain', type: 'short-input', placeholder: 'yoursite.atlassian.net', required: true, }, { id: 'projectSelector', title: 'Project', type: 'selector', selectorKey: 'jira.projects', canonicalParamId: 'projectKey', mode: 'basic', dependsOn: ['domain'], placeholder: 'Select a project', required: true, }, { id: 'projectKey', title: 'Project Key', type: 'short-input', canonicalParamId: 'projectKey', mode: 'advanced', placeholder: 'e.g. ENG, PROJ', required: true, }, ] ``` ### How `dependsOn` maps to `SelectorContext` The shared connector context builder projects only active dependencies. A canonical dependency uses its active basic or advanced value under `canonicalParamId`; a non-canonical dependency uses its field `id`. The resulting key must be a `SelectorContextKey` in `apps/sim/lib/selectors/types.ts` and must be explicitly allowed by that selector's manifest entry. The browser sends the connector's workspace scope, not the complete connector configuration. ### Available selector keys Check `apps/sim/lib/selectors/manifest.ts` for the exhaustive selector keys. Common ones for connectors: | SelectorKey | Context Deps | Returns | |-------------|-------------|---------| | `airtable.bases` | credential | Base ID + name | | `airtable.tables` | credential, `baseId` | Table ID + name | | `slack.channels` | credential | Channel ID + name | | `gmail.labels` | credential | Label ID + name | | `google.calendar` | credential | Calendar ID + name | | `linear.teams` | credential | Team ID + name | | `linear.projects` | credential, `teamId` | Project ID + name | | `jira.projects` | credential, `domain` | Project key + name | | `confluence.spaces` | credential, `domain` | Space key + name | | `notion.databases` | credential | Database ID + name | | `asana.workspaces` | credential | Workspace GID + name | | `microsoft.teams` | credential | Team ID + name | | `microsoft.channels` | credential, `teamId` | Channel ID + name | | `webflow.sites` | credential | Site ID + name | | `outlook.folders` | credential | Folder ID + name | ## ExternalDocument Shape Every document returned from `listDocuments`/`getDocument` must include: ```typescript { externalId: string // Source-specific unique ID title: string // Document title content: string // Extracted plain text (or '' if contentDeferred) contentDeferred?: boolean // true = content will be fetched via getDocument