Add or update a Sim knowledge base connector for syncing documents from an external source, including auth mode, config fields, pagination, document mapping, tags, and registry wiring. Use when working in `apps/sim/connectors/{service}/` or adding a new external document source.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Instruções da origem · Visualização somente leitura
name
add-connector
description
Add or update a Sim knowledge base connector for syncing documents from an external source, including auth mode, config fields, pagination, document mapping, tags, and registry wiring. Use when working in `apps/sim/connectors/{service}/` or adding a new external document source.
argument-hint
<service-name> [api-docs-url]
Add Connector Skill
You are an expert at adding knowledge base connectors to Sim. A connector syncs documents from an external source (Confluence, Google Drive, Notion, etc.) into a knowledge base.
Your Task
When the user asks you to create a connector:
Use Context7 or WebFetch to read the service's API documentation
Determine the auth mode: OAuth (if Sim already has an OAuth provider for the service) or API key (if the service uses API key / Bearer token auth)
Create the connector directory: a client-safe meta.ts (declarative metadata) plus the runtime module that spreads it
Register it in BOTH the server registry and the client-safe meta registry
Hard Rule: No Guessed Response Or Document Schemas
If the service docs do not clearly show the document list response, document fetch response, pagination shape, or metadata fields, you MUST tell the user instead of guessing.
Do NOT invent document fields
Do NOT guess pagination cursors or next-page fields
Do NOT infer metadata/tag mappings from unrelated endpoints
Do NOT fabricate ExternalDocument content structure from partial docs
If the source schema is unknown, do one of these instead:
Ask the user for sample API responses
Ask the user for test credentials so you can verify live payloads
Implement only the documented parts of the connector
Leave the connector incomplete and explicitly say which fields remain unknown
Directory Structure
Each connector is split into a client-safe metadata file and a server-only runtime file. This mirrors the XBlockMeta / BLOCK_META_REGISTRY split in apps/sim/blocks — client components (the knowledge UI) only need the metadata (icon, name, auth, config fields), so the runtime functions (which pull server-only helpers like input-validation.server → undici → node:net) must stay out of the client bundle.
Create files in apps/sim/connectors/{service}/:
connectors/{service}/
├── index.ts # Barrel export (re-exports the runtime connector)
├── meta.ts # ConnectorMeta — client-safe declarative metadata
└── {service}.ts # ConnectorConfig — spreads the meta + adds runtime functions
meta.ts exports {service}ConnectorMeta: ConnectorMeta. It imports ONLY the icon from @/components/icons, import type { ConnectorMeta } from '@/connectors/types', and any pure-data constants. It must NEVER import server/runtime code.
{service}.ts exports {service}Connector: ConnectorConfig. It imports the meta via import { {service}ConnectorMeta } from '@/connectors/{service}/meta', spreads it as the first property, and holds the runtime functions (which may import server-only helpers like @/lib/knowledge/documents/utils).
Authentication
Connectors use a discriminated union for auth config (ConnectorAuthConfig in connectors/types.ts):
For services with existing OAuth providers in apps/sim/lib/oauth/types.ts. The provider must match an OAuthService. The modal shows a credential picker and handles token refresh automatically.
API key mode
For services that use API key / Bearer token auth. The modal shows a password input with the configured label and placeholder. The API key is encrypted at rest using AES-256-GCM and stored in a dedicated encryptedApiKey column on the connector record. The sync engine decrypts it automatically — connectors receive the raw access token in listDocuments, getDocument, and validateConfig.
Connector Structure (meta.ts + runtime)
The declarative metadata lives in meta.ts (ConnectorMeta). The runtime functions live in {service}.ts (ConnectorConfig), which spreads the meta as its first property.
meta.ts — client-safe metadata
import { {Service}Icon } from'@/components/icons'importtype { ConnectorMeta } from'@/connectors/types'exportconst {service}ConnectorMeta: ConnectorMeta = {
id: '{service}',
name: '{Service}',
description: 'Sync documents from {Service} into your knowledge base',
version: '1.0.0',
icon: {Service}Icon,
auth: {
mode: 'oauth',
provider: '{service}', // Must match OAuthService in lib/oauth/types.tsrequiredScopes: ['read:...'],
},
configFields: [
// Rendered dynamically by the add-connector modal UI// Supports 'short-input', 'dropdown', and 'selector' types — see ConfigField Types below
],
// Optional: tag definitions are metadata too — declare them here// tagDefinitions: [ ... ],
}
Keep meta.ts free of any server/runtime import. Only the icon, the ConnectorMeta type, and pure-data constants belong here.
{service}.ts — runtime (OAuth example)
import { createLogger } from'@sim/logger'import { fetchWithRetry } from'@/lib/knowledge/documents/utils'import { {service}ConnectorMeta } from'@/connectors/{service}/meta'importtype { ConnectorConfig, ExternalDocument, ExternalDocumentList } from'@/connectors/types'const logger = createLogger('{Service}Connector')
exportconst {service}Connector: ConnectorConfig = {
...{service}ConnectorMeta,
listDocuments: async (accessToken, sourceConfig, cursor) => {
// Return metadata stubs with contentDeferred: true (if per-doc content fetch needed)// Or full documents with content (if list API returns content inline)// Return { documents: ExternalDocument[], nextCursor?, hasMore }
},
getDocument: async (accessToken, sourceConfig, externalId) => {
// Fetch full content for a single document// Return ExternalDocument with contentDeferred: false, or null
},
validateConfig: async (accessToken, sourceConfig) => {
// Return { valid: true } or { valid: false, error: 'message' }
},
// Optional: map source metadata to semantic tag keys (translated to slots by sync engine)mapTags: (metadata) => {
// Return Record<string, unknown> with keys matching tagDefinitions[].id
},
}
Only map fields in listDocuments, getDocument, validateConfig, and mapTags when the source payload shape is documented or live-verified. If not, tell the user and stop rather than guessing.
API key connector example
The split is identical — auth lives in meta.ts, runtime functions in {service}.ts.
Use type: 'selector' to fetch options dynamically from the existing selector registry (hooks/selectors/registry.ts). Selectors are always paired with a manual fallback input using the canonical pair pattern — a selector field (basic mode) and a short-input field (advanced mode) linked by canonicalParamId.
The user sees a toggle button (ArrowLeftRight) to switch between the selector dropdown and manual text input. On submit, the modal resolves each canonical pair to the active mode's value, keyed by canonicalParamId.
Rules
Every selector field MUST have a canonical pair — a corresponding short-input (or dropdown) field with the same canonicalParamId and mode: 'advanced'.
required must be set identically on both fields in a pair. If the selector is required, the manual input must also be required.
canonicalParamId must match the key the connector expects in sourceConfig (e.g. baseId, channel, teamId). The advanced field's id should typically match canonicalParamId.
dependsOn references the selector field's id, not the canonicalParamId. The modal propagates dependency clearing across canonical siblings automatically — changing either field in a parent pair clears dependent children.
Selector canonical pair example (Airtable base → table cascade)
Selector with domain dependency (Jira/Confluence pattern)
When a selector depends on a plain short-input field (no canonical pair), dependsOn references that field's id directly. The domain field's value maps to SelectorContext.domain automatically via SELECTOR_CONTEXT_FIELDS.
The connector selector field builds a SelectorContext from dependency values. For the mapping to work, each dependency's canonicalParamId (or field id for non-canonical fields) must exist in SELECTOR_CONTEXT_FIELDS (lib/workflows/subblocks/context.ts):
Check hooks/selectors/types.ts for the full SelectorKey union. Common ones for connectors:
SelectorKey
Context Deps
Returns
airtable.bases
credential
Base ID + name
airtable.tables
credential, baseId
Table ID + name
slack.channels
credential
Channel ID + name
gmail.labels
credential
Label ID + name
google.calendar
credential
Calendar ID + name
linear.teams
credential
Team ID + name
linear.projects
credential, teamId
Project ID + name
jira.projects
credential, domain
Project key + name
confluence.spaces
credential, domain
Space key + name
notion.databases
credential
Database ID + name
asana.workspaces
credential
Workspace GID + name
microsoft.teams
credential
Team ID + name
microsoft.channels
credential, teamId
Channel ID + name
webflow.sites
credential
Site ID + name
outlook.folders
credential
Folder ID + name
ExternalDocument Shape
Every document returned from listDocuments/getDocument must include:
{
externalId: string// Source-specific unique IDtitle: string// Document titlecontent: string// Extracted plain text (or '' if contentDeferred)contentDeferred?: boolean// true = content will be fetched via getDocumentmimeType: 'text/plain'// Always text/plain (content is extracted)contentHash: string// Metadata-based hash for change detectionsourceUrl?: string// Link back to original (stored on document record)metadata?: Record<string, unknown> // Source-specific data (fed to mapTags)
}
Content Deferral (Required for file/content-download connectors)
All connectors that require per-document API calls to fetch content MUST use contentDeferred: true. This is the standard pattern — listDocuments returns lightweight metadata stubs, and content is fetched lazily by the sync engine via getDocument only for new/changed documents.
This pattern is critical for reliability: the sync engine processes documents in batches and enqueues each batch for processing immediately. If a sync times out, all previously-batched documents are already queued. Without deferral, content downloads during listing can exhaust the sync task's time budget before any documents are saved.
When to use contentDeferred: true
The service's list API does NOT return document content (only metadata)
Content requires a separate download/export API call per document
The list API already returns the full content inline (e.g., Slack messages, Reddit posts, HubSpot notes)
No per-document API call is needed to get content
Content Hash Strategy
Use a metadata-basedcontentHash — never a content-based hash. The hash must be derivable from the list response metadata alone, so the sync engine can detect changes without downloading content.
Good metadata hash sources:
modifiedTime / lastModifiedDateTime — changes when file is edited
// Google Drive: modifiedTime changes on editcontentHash: `gdrive:${file.id}:${file.modifiedTime ?? ''}`// GitHub: blob SHA is a content-addressable hashcontentHash: `gitsha:${item.sha}`// Dropbox: API provides content_hashcontentHash: `dropbox:${entry.id}:${entry.content_hash ?? entry.server_modified}`// Confluence: version number increments on editcontentHash: `confluence:${page.id}:${page.version.number}`
Critical invariant: The contentHash MUST be identical whether produced by listDocuments (stub) or getDocument (full doc). Both should use the same stub function to guarantee this.
Notion: connectors/notion/notion.ts — blocks API with last_edited_time hash
Confluence: connectors/confluence/confluence.ts — version number hash
tagDefinitions — Declared Tag Definitions
Declare which tags the connector populates using semantic IDs. Shown in the add-connector modal as opt-out checkboxes.
On connector creation, slots are dynamically assigned via getNextAvailableSlot — connectors never hardcode slot names.
id: Semantic key matching a key returned by mapTags (e.g. 'labels', 'version')
displayName: Human-readable name shown in the UI (e.g. "Labels", "Last Modified")
fieldType: 'text' | 'number' | 'date' | 'boolean' — determines which slot pool to draw from
Users can opt out of specific tags in the modal. Disabled IDs are stored in sourceConfig.disabledTagIds.
The assigned mapping (semantic id → slot) is stored in sourceConfig.tagSlotMapping.
@/connectors/utils Helpers
Reuse these instead of inlining the same logic (the validator enforces them):
htmlToPlainText(html) — strip HTML to plain text before indexing ExternalDocument.content. Never index raw HTML.
computeContentHash(content) — stable content hash for change detection.
parseTagDate(value) — parse to a valid Date or undefined (guards Invalid Date). Use in mapTags for date fields.
joinTagArray(value) — validate an array and join to a comma-separated string, or undefined. Use in mapTags for array/label fields.
parseMultiValue(value) — normalize a value into a string[].
mapTags — Metadata to Semantic Keys
Maps source metadata to semantic tag keys. Required if tagDefinitions is set.
The sync engine calls this automatically and translates semantic keys to actual DB slots
using the tagSlotMapping stored on the connector.
Return keys must match the id values declared in tagDefinitions.
Use the @/connectors/utils helpers for the common transforms — don't hand-roll date/array validation:
import { joinTagArray, parseTagDate } from'@/connectors/utils'mapTags: (metadata: Record<string, unknown>): Record<string, unknown> => {
constresult: Record<string, unknown> = {}
// joinTagArray validates the array and joins to a comma-separated string (undefined if empty)const labels = joinTagArray(metadata.labels)
if (labels) result.labels = labels
// Validate numbers — guard against NaNif (metadata.version != null) {
const num = Number(metadata.version)
if (!Number.isNaN(num)) result.version = num
}
// parseTagDate returns a valid Date or undefined (guards against Invalid Date)const lastModified = parseTagDate(metadata.lastModified)
if (lastModified) result.lastModified = lastModified
return result
}
External API Calls — Use fetchWithRetry
All external API calls must use fetchWithRetry from @/lib/knowledge/documents/utils instead of raw fetch(). This provides exponential backoff with retries on 429/502/503/504 errors. It returns a standard Response — all .ok, .json(), .text() checks work unchanged.
For validateConfig (user-facing, called on save), pass VALIDATE_RETRY_OPTIONS to cap wait time at ~7s. Background operations (listDocuments, getDocument) use the built-in defaults (5 retries, ~31s max).
If ExternalDocument.sourceUrl is set, the sync engine stores it on the document record. Always construct the full URL (not a relative path).
Capped or Incomplete Listings — syncContext.listingCapped (REQUIRED)
If listDocuments can ever return less than the full source set on a non-incremental sync — a maxItems/maxDocuments-style cap, or a transient per-item error that drops a still-existing document from the listing — it MUST set syncContext.listingCapped = true when that happens.
The sync engine reconciles deletions by comparing the full listing against stored documents: anything not seen is hard-deleted (sync-engine.ts, gated on !syncContext?.listingCapped). A truncated listing without this flag deletes every real document beyond the cap. This was the single most common bug found when auditing connectors — do not omit it.
if (hitLimit && syncContext) {
syncContext.listingCapped = true
}
Rules:
Set it when a user-configured cap truncates the listing while more documents exist
Set it when a thrown error caused a still-present document to be skipped during listing
Do NOT set it when the source is genuinely exhausted (deleted documents must still reconcile)
Do NOT set it for intentional scope filters (e.g. a date cutoff) — out-of-scope documents should be reconciled normally
Sync Engine Behavior (Do Not Modify)
The sync engine (lib/knowledge/connectors/sync-engine.ts) is connector-agnostic. It:
Calls listDocuments with pagination until hasMore is false
Compares contentHash to detect new/changed/unchanged documents
Stores sourceUrl and calls mapTags on insert/update automatically
Handles soft-delete of removed documents
Resolves access tokens automatically — OAuth tokens are refreshed, API keys are decrypted from the encryptedApiKey column
You never need to modify the sync engine when adding a connector.
Icon
The icon field on ConnectorConfig is used throughout the UI — in the connector list, the add-connector modal, and as the document icon in the knowledge base table (replacing the generic file type icon for connector-sourced documents). The icon is read from CONNECTOR_REGISTRY[connectorType].icon at runtime — no separate icon map to maintain.
If the service already has an icon in apps/sim/components/icons.tsx (from a tool integration), reuse it. Otherwise, ask the user to provide the SVG.
Registering
Register in BOTH registries, keeping the same alphabetical-by-id ordering in each.
Server registry — apps/sim/connectors/registry.server.ts (server-only full registry; holds full connectors with runtime functions, imported by the sync engine and knowledge API routes):
Client-safe meta registry — apps/sim/connectors/registry.ts (imports each connector's meta.ts only, so client components can use it without pulling server-only code; the metadata counterpart to BLOCK_META_REGISTRY):
registry.ts exports CONNECTOR_META_REGISTRY: ConnectorMetaRegistry plus the helpers getConnectorMeta(id) and getAllConnectorMeta(), importing each @/connectors/{service}/meta directly — never the runtime module. registry.server.ts exports CONNECTOR_REGISTRY: ConnectorRegistry.
Reference Implementations
OAuth + contentDeferred: apps/sim/connectors/google-drive/google-drive.ts — file download with metadata-based hash, orderBy for deterministic pagination
API key: apps/sim/connectors/fireflies/fireflies.ts — GraphQL API with Bearer token auth
Checklist
Created connectors/{service}/meta.ts with {service}ConnectorMeta: ConnectorMeta (icon, name, auth, configFields, tagDefinitions) — no server/runtime imports
Created connectors/{service}/{service}.ts with {service}Connector: ConnectorConfig spreading the meta + runtime functions
Created connectors/{service}/index.ts barrel export
Auth configured correctly:
OAuth: auth.provider matches an existing OAuthService in lib/oauth/types.ts
API key: auth.label and auth.placeholder set appropriately
Every type: 'selector' field has a canonical pair (short-input or dropdown with same canonicalParamId and mode: 'advanced')
required is identical on both fields in each canonical pair
selectorKey exists in hooks/selectors/registry.ts
dependsOn references selector field IDs (not canonicalParamId)
Dependency canonicalParamId values exist in SELECTOR_CONTEXT_FIELDS
listDocuments handles pagination with metadata-based content hashes
syncContext.listingCapped = true set whenever the listing is truncated (max-items cap or transient per-item error) — required to prevent the engine's deletion reconciliation from removing unseen documents
contentDeferred: true used if content requires per-doc API calls (file download, export, blocks fetch)
contentHash is metadata-based (not content-based) and identical between stub and getDocument
sourceUrl set on each ExternalDocument (full URL, not relative)
metadata includes source-specific data for tag mapping
tagDefinitions declared for each semantic key returned by mapTags
mapTags implemented if source has useful metadata (labels, dates, versions)
validateConfig verifies the source is accessible
All external API calls use fetchWithRetry (not raw fetch)
All optional config fields validated in validateConfig
Icon exists in components/icons.tsx (or asked user to provide SVG)
Registered the full connector in connectors/registry.server.ts
Registered the meta in connectors/registry.ts (same alphabetical-by-id ordering as registry.server.ts)
