validate-integration

# Validate Integration Skill You are an expert auditor for Sim integrations. Your job is to thoroughly validate that an existing integration is correct, complete, and follows all conventions. ## Your Task When the user asks you to validate an integration: 1. Read the service's API documentation (via WebFetch or Context7) 2. Read every tool, the block, and registry entries 3. Cross-reference everything against the API docs and Sim conventions 4. Report all issues found, grouped by severity (critical, warning, suggestion) 5. Fix all issues after reporting them ## Step 1: Gather All Files Read **every** file for the integration — do not skip any: ``` apps/sim/tools/{service}/ # All tool files, types.ts, index.ts apps/sim/blocks/blocks/{service}.ts # Block definition apps/sim/tools/registry.ts # Tool registry entries for this service apps/sim/blocks/registry-maps.ts # Block + meta registry entry (BLOCK_REGISTRY / BLOCK_META_REGISTRY) apps/sim/components/icons.tsx # Icon definition apps/sim/lib/auth/connectors/providers.ts # Better Auth connector providers (buildConnectorProviders) — should use getCanonicalScopesForProvider() apps/sim/lib/oauth/oauth.ts # OAuth provider config — single source of truth for scopes apps/sim/lib/oauth/utils.ts # Scope utilities, SCOPE_DESCRIPTIONS for modal UI packages/deployment-config/src/env-capabilities.ts # OAuth client runtime capability source of truth apps/sim/lib/core/config/env.ts # Runtime env schema for capability fields packages/sim-setup/src/capability-config.ts # Exhaustive CLI input-mode mapping for OAuth fields packages/deployment-config/src/integrations.json # Generated client-safe integration catalog packages/deployment-config/src/service-account-providers.generated.ts # Generated provider-ID facts packages/deployment-config/src/service-account-metadata.ts # Handwritten deployment policy ``` If the block, its triggers, or connector fields use a `selectorKey`, also apply the `validate-selector` skill and read the key's entry in `apps/sim/lib/selectors/manifest.ts`, its server attachment and provider listing primitive, and the shared context builder. Selector provider logic runs only server-side through `selectors.execute`. ## Step 2: Pull API Documentation Fetch the official API docs for the service. This is the **source of truth** for: - Endpoint URLs, HTTP methods, and auth headers - Required vs optional parameters - Parameter types and allowed values - Response shapes and field names - Pagination patterns (which param name, which response field) - Rate limits and error formats ### Hard Rule: No Guessed Response Schemas If the official docs do not clearly show the response JSON shape for an endpoint, you MUST tell the user instead of guessing. - Do NOT assume field names from nearby endpoints - Do NOT infer nested JSON paths without evidence - Do NOT treat "likely" fields as confirmed outputs - Do NOT accept implementation guesses as valid just because they are defensive If a response schema is unknown, the validation must explicitly call that out and require: 1. sample responses from the user, 2. live test credentials for verification, or 3. trimming the tool/block down to only documented fields. ## Step 3: Validate Tools For **every** tool file, check: ### Tool ID and Naming - [ ] Tool ID uses `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`, `slack_send_message`) - [ ] Tool `name` is human-readable (e.g., `'X Create Tweet'`) - [ ] Tool `description` is a concise one-liner describing what it does - [ ] Tool `version` is set (`'1.0.0'` or `'2.0.0'` for V2) ### Params - [ ] All required API params are marked `required: true` - [ ] All optional API params are marked `required: false` - [ ] Every param has explicit `required: true` or `required: false` — never omitted - [ ] Param types match the API (`'string'`, `'number'`, `'boolean'`, `'json'`) - [ ] Visibility is correct: - `'hidden'` — ONLY for OAuth access tokens and system-injected params - `'user-only'` — for API keys, credentials, and account-specific IDs the user must provide - `'user-or-llm'` — for everything else (search queries, content, filters, IDs that could come from other blocks) - [ ] Every param has a `description` that explains what it does ### Request - [ ] URL matches the API endpoint exactly (correct base URL, path segments, path params) - [ ] HTTP method matches the API spec (GET, POST, PUT, PATCH, DELETE) - [ ] Headers include correct auth pattern: - OAuth: `Authorization: Bearer ${params.accessToken}` - API Key: correct header name and format per the service's docs - [ ] `Content-Type` header is set for POST/PUT/PATCH requests - [ ] Body sends all required fields and only includes optional fields when provided - [ ] For GET requests with query params: URL is constructed correctly with query string - [ ] ID fields in URL paths are `.trim()`-ed to prevent copy-paste whitespace errors - [ ] Path params use template literals correctly: `` `https://api.service.com/v1/${params.id.trim()}` `` ### Response / transformResponse - [ ] Correctly parses the API response (`await response.json()`) - [ ] Extracts the right fields from the response structure (e.g., `data.data` vs `data` vs `data.results`) - [ ] All nullable fields use `?? null` - [ ] All optional arrays use `?? []` - [ ] Error cases are handled: checks for missing/empty data and returns meaningful error - [ ] Does NOT do raw JSON dumps — extracts meaningful, individual fields - [ ] Every extracted field is backed by official docs or live-verified sample payloads ### Outputs - [ ] All output fields match what the API actually returns - [ ] No fields are missing that the API provides and users would commonly need - [ ] No phantom fields defined that the API doesn't return - [ ] `optional: true` is set on fields that may not exist in all responses - [ ] When using `type: 'json'` and the shape is known, `properties` defines the inner fields (tool outputs only — block outputs do not support `properties`) - [ ] When using `type: 'array'`, `items` defines the item structure with `properties` (tool outputs only) - [ ] Field descriptions are accurate and helpful ### Types (types.ts) - [ ] Has param interfaces for every tool (e.g., `XCreateTweetParams`) - [ ] Has response interfaces for every tool (extending `ToolResponse`) - [ ] Optional params use `?` in the interface (e.g., `replyTo?: string`) - [ ] Field names in types match actual API field names - [ ] Shared response types are properly reused (e.g., `XTweetResponse` shared across tweet tools) ### Barrel Export (index.ts) - [ ] Every tool is exported - [ ] All types are re-exported (`export * from './types'`) - [ ] No orphaned exports (tools that don't exist) ### Tool Registry (tools/registry.ts) - [ ] Every tool is imported and registered - [ ] Registry keys use snake_case and match tool IDs exactly - [ ] Entries are in alphabetical order within the file ### Resolved-Secret Provenance and Model Input For every request field, determine whether it is ordinary API input, model-visible text/structured content, opaque model input, or a value persisted into Sim-owned durable storage. Treat model-input provenance as opt-in. Require official documentation or an unambiguous local execution path proving that the exact field reaches an AI model. If the evidence is ambiguous, leave the integration unchanged; do not infer a model boundary merely from natural-language, search, extraction, or "AI-powered" marketing terminology. - [ ] AI-consumed text/structured fields use `request.modelInput` with `mode: 'project'` and a minimal exact selector; nested/JSON-string adapters preserve shape through `applyProjected` - [ ] Ordinary external URLs, domains, resource IDs, and control fields retain normal request semantics unless the exact field is proven model-visible; an AI-backed provider or later model processing of the referenced resource is not sufficient evidence - [ ] Serialized content proven to be sent directly to an external model is selected by `request.modelInput`, projected before the existing formatter parses it, and has deterministic formatter behavior when a whole-value placeholder is invalid for the serialized grammar - [ ] Actual inline/raw AI-consumed bytes owned by a registered in-process operation use `operation.modelInput` with `privateInputPaths` (or `mode: 'private-provenance'`), and the operation calls `validateOpaqueModelInputProvenance` before model egress; storage keys, paths, signed URLs, and ordinary remote URLs are not treated as byte provenance, while tracked stored bytes are authorized independently at the owning model-egress boundary - [ ] Persisted workspace-file contents are checked with the shared provenance guard only when their bytes or decoded content cross into a model/tool-result boundary; ordinary file APIs remain unchanged. Unsupported secret-bearing file paths are rejected at `file_write` - [ ] Sim-owned durable writes and internal execution handoffs that can enter workflows/models use field-scoped `operation.secretProvenance`; the owning operation validates the exact selection and scope, strip private metadata, and persist, import, or propagate it at the owning boundary - [ ] Private provenance is never attached to external URLs; registered in-process operations preserve it through `operation.modelInput` / `operation.secretProvenance`, while proven model-visible external fields use request projection and other external inputs remain unchanged - [ ] No tool performs raw secret plaintext/source substitution or serializes plaintext provenance - [ ] No `transformResponse` or tool-local helper blanket-sanitizes ordinary third-party results; only execution-scoped, activated Sim provenance is projected at shared model/log boundaries - [ ] Private headers/envelopes are produced and stripped by the shared tool executor, never hand-rolled or returned as functional output - [ ] Every added provenance hook has a concrete Sim `{{...}}` resolution path and a later persistence/model/log crossing; there is no generic handling for arbitrary filenames, metadata, provider results, or API payloads - [ ] Diagnostic projection is applied only to values carrying execution-scoped provenance; ordinary provider responses, filenames, URLs, and errors are unchanged - [ ] Tests cover named `{{NAME}}` projection, unproven identical public text, nested and serialized shape handling, unchanged ordinary external inputs, malformed/incomplete metadata, headerless legacy requests, metadata stripping, and durable legacy/stale/scope cases when applicable Treat a missing or bypassed model, durable, or internal-execution provenance boundary as **critical**. Do not fix it with a tool-specific string replacer or by sanitizing every provider result; repair the shared request, in-process operation, persistence, or re-entry boundary that owns the data. ## Step 4: Validate Block ### Block ↔ Tool Alignment (CRITICAL) This is the most important validation — the block must be perfectly aligned with every tool it references. For **each tool** in `tools.access`: - [ ] The operation dropdown has an option whose ID matches the tool ID (or the `tools.config.tool` function correctly maps to it) - [ ] Every **required** tool param (except `accessToken`) has a corresponding subBlock input that is: - Shown when that operation is selected (correct `condition`) - Marked as `required: true` (or conditionally required) - [ ] Every **optional** tool param has a corresponding subBlock input (or is intentionally omitted if truly never needed) - [ ] Every subBlock `id` is unique (duplicates collide silently; the last definition wins). `blocks.test.ts` fails a duplicate within one condition unless the copies are a basic/advanced mode-swap pair, one basic plus trigger-mode copies, or all carry `canonicalParamId`. The only sanctioned cross-condition reuse is the hosted-key `apiKey` pair (`add-hosted-key` skill) - [ ] The `tools.config.tool` function returns the correct tool ID for every possible operation value - [ ] Each subBlock (or its `canonicalParamId`) is named exactly after the tool param it fills. A required `user-only` param that is only renamed in `tools.config.params` fails `bun run apps/sim/scripts/check-block-registry.ts origin/staging`; remap only optional or `user-or-llm` params ### SubBlocks - [ ] Operation dropdown lists ALL tool operations available in `tools.access` - [ ] Dropdown option labels are human-readable and descriptive - [ ] Conditions use correct syntax: - Single value: `{ field: 'operation', value: 'x_create_tweet' }` - Multiple values (OR): `{ field: 'operation', value: ['x_create_tweet', 'x_delete_tweet'] }` - Negation: `{ field: 'operation', value: 'delete', not: true }` - Compound: `{ field: 'op', value: 'send', and: { field: 'type', value: 'dm' } }` - [ ] Condition arrays include ALL operations that use that field — none missing - [ ] `dependsOn` is set for fields that need other values (selectors depending on credential, cascading dropdowns) - [ ] SubBlock types match tool param types: - Enum/fixed options → `dropdown` - Free text → `short-input` - Long text/content → `long-input` - True/false → `switch` (a Yes/No `dropdown` only when the tool needs a third "unset" state) - Credentials → `oauth-input` with correct `serviceId` - [ ] Dropdown `value: () => 'default'` is set for dropdowns with a sensible default - [ ] Every `short-input`, `long-input`, `code`, and selector subBlock has a `placeholder` — including password fields (`Enter your API key`). Formatted values show the shape (`2023-01-01T00:00:00Z`); optional fields with a server default name it. See add-integration → Step 3 ### Advanced Mode - [ ] Optional, rarely-used fields are set to `mode: 'advanced'`: - Pagination tokens / next tokens - Time range filters (start/end time) - Sort order / direction options - Max results / per page limits - Reply settings / threading options - Rarely used IDs (reply-to, quote-tweet, etc.) - Exclude filters - [ ] **Required** fields are NEVER set to `mode: 'advanced'` - [ ] Fields that users fill in most of the time are NOT set to `mode: 'advanced'` ### WandConfig - [ ] Timestamp fields have `wandConfig` with `generationType: 'timestamp'` - [ ] Comma-separated list fields have `wandConfig` with a descriptive prompt - [ ] Complex filter/query fields have `wandConfig` with format examples in the prompt - [ ] All `wandConfig` prompts end with an explicit `Return ONLY the <format>` instruction so the generated value can be pasted directly into the field - [ ] `wandConfig.placeholder` describes what to type in natural language ### Tools Config - [ ] `tools.access` lists **every** tool ID the block can use — none missing - [ ] `tools.config.tool` returns the correct tool ID for each operation - [ ] Type coercions are in `tools.config.params` (runs at execution time), NOT in `tools.config.tool` (runs at serialization time before variable resolution — coercing there destroys dynamic references like `<Block.output>`) - [ ] `tools.config.params` handles: - `Number()` conversion for numeric params that come as strings from inputs - `Boolean` / string-to-boolean conversion for toggle params - Empty string → `undefined` conversion for optional dropdown values - SubBlock ID → tool param name remapping only for optional or `user-or-llm` params ### Block Outputs - [ ] Outputs cover the key fields returned by ALL tools (not just one operation) - [ ] Output types are correct (`'string'`, `'number'`, `'boolean'`, `'json'`, `'file'`, `'file[]'`) - [ ] `type: 'json'` outputs describe inner fields in the description string: `'User profile (id, name, username, bio)'` or `'[{address, status, type}]'` for arrays - [ ] **Do NOT add a `properties: {...}` field on block outputs.** Block-level `OutputFieldDefinition` (from `@sim/workflow-types/blocks`) only accepts `{ type, description?, condition?, hiddenFromDisplay? }`. Nested `properties` is a tool-level construct (`OutputProperty`) — adding it to a block output will fail TypeScript at build time - [ ] No opaque `type: 'json'` with vague descriptions like `'Response data'` - [ ] Outputs that only appear for certain operations use `condition` if supported, or document which operations return them ### Block Metadata - [ ] `type` is snake_case (e.g., `'x'`, `'cloudflare'`) - [ ] `name` is human-readable (e.g., `'X'`, `'Cloudflare'`) - [ ] `description` is a concise one-liner - [ ] `longDescription` provides detail for docs - [ ] `docsLink` points to `'https://docs.sim.ai/integrations/{service}'` - [ ] `category` is `'tools'` - [ ] `bgColor` uses the service's brand color hex - [ ] `icon` references the correct icon component from `@/components/icons` - [ ] `authMode` is set correctly (`AuthMode.OAuth` or `AuthMode.ApiKey`) - [ ] Block + meta are registered in `blocks/registry-maps.ts` (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`) alphabetically ### BlockMeta - [ ] `{Service}BlockMeta` is exported in the same file as the block - [ ] Has at least 7 templates, each with `icon`, `title`, `prompt`, `modules`, `category`, and `tags` - [ ] Prompts describe concrete use cases, not generic descriptions of what the service does - [ ] `alsoIntegrations` is set on any template whose prompt references another service - [ ] `skills` present (3–5 mainstream, 2–3 niche), each grounded in `tools.access` — flag any skill implying an unsupported action - [ ] **Each skill is real, not hallucinated** — web-search and confirm it maps to a popular use case attested online (vendor use-case pages, official docs describing the workflow, reputable "top automations" articles); rewrite/remove any you cannot source - [ ] Each skill has a kebab-case `name` (≤64 chars, unique), a one-line `description`, and markdown `content` with `# Title` + `## Steps` + an output/guidance section ### Block Inputs