validate-integration

name	validate-integration
description	Validate an existing Sim integration (tools, block, registry, and resolved-secret/model-input boundaries) against the service's API docs and Sim execution conventions
argument-hint	<service-name> [api-docs-url]

Validate Integration Skill

You are an expert auditor for Sim integrations. Your job is to thoroughly validate that an existing integration is correct, complete, and follows all conventions.

Your Task

When the user asks you to validate an integration:

1. Read the service's API documentation (via WebFetch or Context7)
2. Read every tool, the block, and registry entries
3. Cross-reference everything against the API docs and Sim conventions
4. Report all issues found, grouped by severity (critical, warning, suggestion)
5. Fix all issues after reporting them

Step 1: Gather All Files

Read every file for the integration — do not skip any:

apps/sim/tools/{service}/          # All tool files, types.ts, index.ts
apps/sim/blocks/blocks/{service}.ts # Block definition
apps/sim/tools/registry.ts          # Tool registry entries for this service
apps/sim/blocks/registry-maps.ts    # Block + meta registry entry (BLOCK_REGISTRY / BLOCK_META_REGISTRY)
apps/sim/components/icons.tsx        # Icon definition
apps/sim/lib/auth/auth.ts           # OAuth config — should use getCanonicalScopesForProvider()
apps/sim/lib/oauth/oauth.ts         # OAuth provider config — single source of truth for scopes
apps/sim/lib/oauth/utils.ts               # Scope utilities, SCOPE_DESCRIPTIONS for modal UI
apps/sim/lib/core/config/env-capabilities.ts # OAuth client runtime capability source of truth
apps/sim/lib/core/config/env.ts     # Runtime env schema for capability fields
scripts/setup/capability-config.ts  # Exhaustive CLI input-mode mapping for OAuth fields
apps/sim/lib/integrations/integrations.json # Generated client-safe integration catalog
apps/sim/lib/integrations/service-account-metadata.ts # Lightweight service-account projection

Step 2: Pull API Documentation

Fetch the official API docs for the service. This is the source of truth for:

• Endpoint URLs, HTTP methods, and auth headers
• Required vs optional parameters
• Parameter types and allowed values
• Response shapes and field names
• Pagination patterns (which param name, which response field)
• Rate limits and error formats

Hard Rule: No Guessed Response Schemas

If the official docs do not clearly show the response JSON shape for an endpoint, you MUST tell the user instead of guessing.

• Do NOT assume field names from nearby endpoints
• Do NOT infer nested JSON paths without evidence
• Do NOT treat "likely" fields as confirmed outputs
• Do NOT accept implementation guesses as valid just because they are defensive

If a response schema is unknown, the validation must explicitly call that out and require:

1. sample responses from the user,
2. live test credentials for verification, or
3. trimming the tool/block down to only documented fields.

Step 3: Validate Tools

For every tool file, check:

Tool ID and Naming

• Tool ID uses snake_case: {service}_{action} (e.g., x_create_tweet, slack_send_message)
• Tool name is human-readable (e.g., 'X Create Tweet')
• Tool description is a concise one-liner describing what it does
• Tool version is set ('1.0.0' or '2.0.0' for V2)

Params

• All required API params are marked required: true
• All optional API params are marked required: false
• Every param has explicit required: true or required: false — never omitted
• Param types match the API ('string', 'number', 'boolean', 'json')
• Visibility is correct:
  • 'hidden' — ONLY for OAuth access tokens and system-injected params
  • 'user-only' — for API keys, credentials, and account-specific IDs the user must provide
  • 'user-or-llm' — for everything else (search queries, content, filters, IDs that could come from other blocks)
• Every param has a description that explains what it does

Request

• URL matches the API endpoint exactly (correct base URL, path segments, path params)
• HTTP method matches the API spec (GET, POST, PUT, PATCH, DELETE)
• Headers include correct auth pattern:
  • OAuth: Authorization: Bearer ${params.accessToken}
  • API Key: correct header name and format per the service's docs
• Content-Type header is set for POST/PUT/PATCH requests
• Body sends all required fields and only includes optional fields when provided
• For GET requests with query params: URL is constructed correctly with query string
• ID fields in URL paths are .trim()-ed to prevent copy-paste whitespace errors
• Path params use template literals correctly: `https://api.service.com/v1/${params.id.trim()}`

Response / transformResponse

• Correctly parses the API response (await response.json())
• Extracts the right fields from the response structure (e.g., data.data vs data vs data.results)
• All nullable fields use ?? null
• All optional arrays use ?? []
• Error cases are handled: checks for missing/empty data and returns meaningful error
• Does NOT do raw JSON dumps — extracts meaningful, individual fields
• Every extracted field is backed by official docs or live-verified sample payloads

Outputs

• All output fields match what the API actually returns
• No fields are missing that the API provides and users would commonly need
• No phantom fields defined that the API doesn't return
• optional: true is set on fields that may not exist in all responses
• When using type: 'json' and the shape is known, properties defines the inner fields (tool outputs only — block outputs do not support properties)
• When using type: 'array', items defines the item structure with properties (tool outputs only)
• Field descriptions are accurate and helpful

Types (types.ts)

• Has param interfaces for every tool (e.g., XCreateTweetParams)
• Has response interfaces for every tool (extending ToolResponse)
• Optional params use ? in the interface (e.g., replyTo?: string)
• Field names in types match actual API field names
• Shared response types are properly reused (e.g., XTweetResponse shared across tweet tools)

Barrel Export (index.ts)

• Every tool is exported
• All types are re-exported (export * from './types')
• No orphaned exports (tools that don't exist)

Tool Registry (tools/registry.ts)

• Every tool is imported and registered
• Registry keys use snake_case and match tool IDs exactly
• Entries are in alphabetical order within the file

Resolved-Secret Provenance and Model Input

For every request field, determine whether it is ordinary API input, model-visible text/structured content, opaque model input, or a value persisted into Sim-owned durable storage.

Treat model-input provenance as opt-in. Require official documentation or an unambiguous local execution path proving that the exact field reaches an AI model. If the evidence is ambiguous, leave the integration unchanged; do not infer a model boundary merely from natural-language, search, extraction, or "AI-powered" marketing terminology.

• AI-consumed text/structured fields use request.modelInput with mode: 'project' and a minimal exact selector; nested/JSON-string adapters preserve shape through applyProjected
• Ordinary external URLs, domains, resource IDs, and control fields retain normal request semantics unless the exact field is proven model-visible; an AI-backed provider or later model processing of the referenced resource is not sufficient evidence
• Serialized content proven to be sent directly to an external model is selected by request.modelInput, projected before the existing formatter parses it, and has deterministic formatter behavior when a whole-value placeholder is invalid for the serialized grammar
• Actual inline/raw AI-consumed bytes owned by an authenticated internal route use privateProvenance (or mode: 'private-provenance'), and the route validates validateOpaqueModelInputProvenance before model egress; storage keys, paths, signed URLs, and ordinary remote URLs are not treated as byte provenance, while tracked stored bytes are authorized independently at the owning model-egress boundary
• Persisted workspace-file contents are checked with the shared provenance guard only when their bytes or decoded content cross into a model/tool-result boundary; ordinary file APIs remain unchanged. Unsupported secret-bearing file paths are rejected at file_write
• Sim-owned durable writes and internal execution handoffs that can enter workflows/models use field-scoped request.secretProvenance; authenticated receivers validate the exact selection and scope, strip private metadata, and persist, import, or propagate it at the owning boundary
• Private provenance is never attached to external URLs or directExecution; proven model-visible external fields use projection, while other external inputs remain unchanged
• No tool performs raw secret plaintext/source substitution or serializes plaintext provenance
• No transformResponse or tool-local helper blanket-sanitizes ordinary third-party results; only execution-scoped, activated Sim provenance is projected at shared model/log boundaries

Treat a missing or bypassed model, durable, or internal-execution provenance boundary as critical. Do not fix it with a tool-specific string replacer or by sanitizing every provider result; repair the shared request, authenticated internal-route, persistence, or re-entry boundary that owns the data.

Step 4: Validate Block

Block ↔ Tool Alignment (CRITICAL)

This is the most important validation — the block must be perfectly aligned with every tool it references.

For each tool in tools.access:

• The operation dropdown has an option whose ID matches the tool ID (or the tools.config.tool function correctly maps to it)
• Every required tool param (except accessToken) has a corresponding subBlock input that is:
  • Shown when that operation is selected (correct condition)
  • Marked as required: true (or conditionally required)
• Every optional tool param has a corresponding subBlock input (or is intentionally omitted if truly never needed)
• SubBlock id values are unique across the entire block — no duplicates even across different conditions
• The tools.config.tool function returns the correct tool ID for every possible operation value
• The tools.config.params function correctly maps subBlock IDs to tool param names when they differ

SubBlocks

• Operation dropdown lists ALL tool operations available in tools.access
• Dropdown option labels are human-readable and descriptive
• Conditions use correct syntax:
  • Single value: { field: 'operation', value: 'x_create_tweet' }
  • Multiple values (OR): { field: 'operation', value: ['x_create_tweet', 'x_delete_tweet'] }
  • Negation: { field: 'operation', value: 'delete', not: true }
  • Compound: { field: 'op', value: 'send', and: { field: 'type', value: 'dm' } }
• Condition arrays include ALL operations that use that field — none missing
• dependsOn is set for fields that need other values (selectors depending on credential, cascading dropdowns)
• SubBlock types match tool param types:
  • Enum/fixed options → dropdown
  • Free text → short-input
  • Long text/content → long-input
  • True/false → dropdown with Yes/No options (not switch unless purely UI toggle)
  • Credentials → oauth-input with correct serviceId
• Dropdown value: () => 'default' is set for dropdowns with a sensible default

Advanced Mode

• Optional, rarely-used fields are set to mode: 'advanced':
  • Pagination tokens / next tokens
  • Time range filters (start/end time)
  • Sort order / direction options
  • Max results / per page limits
  • Reply settings / threading options
  • Rarely used IDs (reply-to, quote-tweet, etc.)
  • Exclude filters
• Required fields are NEVER set to mode: 'advanced'
• Fields that users fill in most of the time are NOT set to mode: 'advanced'

WandConfig

• Timestamp fields have wandConfig with generationType: 'timestamp'
• Comma-separated list fields have wandConfig with a descriptive prompt
• Complex filter/query fields have wandConfig with format examples in the prompt
• All wandConfig prompts end with "Return ONLY the [format] - no explanations, no extra text."
• wandConfig.placeholder describes what to type in natural language

Tools Config

• tools.access lists every tool ID the block can use — none missing
• tools.config.tool returns the correct tool ID for each operation
• Type coercions are in tools.config.params (runs at execution time), NOT in tools.config.tool (runs at serialization time before variable resolution)
• tools.config.params handles:
  • Number() conversion for numeric params that come as strings from inputs
  • Boolean / string-to-boolean conversion for toggle params
  • Empty string → undefined conversion for optional dropdown values
  • Any subBlock ID → tool param name remapping
• No Number(), JSON.parse(), or other coercions in tools.config.tool — these would destroy dynamic references like <Block.output>

Block Outputs

• Outputs cover the key fields returned by ALL tools (not just one operation)
• Output types are correct ('string', 'number', 'boolean', 'json')
• type: 'json' outputs describe inner fields in the description string: 'User profile (id, name, username, bio)' or '[{address, status, type}]' for arrays
• Do NOT add a properties: {...} field on block outputs. Block-level OutputFieldDefinition (from @sim/workflow-types/blocks) only accepts { type, description?, condition?, hiddenFromDisplay? }. Nested properties is a tool-level construct (OutputProperty) — adding it to a block output will fail TypeScript at build time
• No opaque type: 'json' with vague descriptions like 'Response data'
• Outputs that only appear for certain operations use condition if supported, or document which operations return them

Block Metadata

• type is snake_case (e.g., 'x', 'cloudflare')
• name is human-readable (e.g., 'X', 'Cloudflare')
• description is a concise one-liner
• longDescription provides detail for docs
• docsLink points to 'https://docs.sim.ai/integrations/{service}'
• category is 'tools'
• bgColor uses the service's brand color hex
• icon references the correct icon component from @/components/icons
• authMode is set correctly (AuthMode.OAuth or AuthMode.ApiKey)
• Block + meta are registered in blocks/registry-maps.ts (BLOCK_REGISTRY / BLOCK_META_REGISTRY) alphabetically

BlockMeta

• {Service}BlockMeta is exported in the same file as the block
• Has at least 7 templates, each with icon, title, prompt, modules, category, and tags
• Prompts describe concrete use cases, not generic descriptions of what the service does
• alsoIntegrations is set on any template whose prompt references another service
• skills present (3–5 mainstream, 2–3 niche), each grounded in tools.access — flag any skill implying an unsupported action
• Each skill is real, not hallucinated — web-search and confirm it maps to a popular use case attested online (vendor use-case pages, official docs describing the workflow, reputable "top automations" articles); rewrite/remove any you cannot source
• Each skill has a kebab-case name (≤64 chars, unique), a one-line description, and markdown content with # Title + ## Steps + an output/guidance section

Block Inputs

• inputs section lists all subBlock params that the block accepts
• Input types match the subBlock types
• When using canonicalParamId, inputs list the canonical ID (not the raw subBlock IDs)

Step 5: Validate OAuth Scopes (if OAuth service)

Scopes are centralized — the single source of truth is OAUTH_PROVIDERS in lib/oauth/oauth.ts.

• Scopes defined in lib/oauth/oauth.ts under OAUTH_PROVIDERS[provider].services[service].scopes
• auth.ts uses getCanonicalScopesForProvider(providerId) — NOT a hardcoded array
• Block requiredScopes uses getScopesForService(serviceId) — NOT a hardcoded array
• No hardcoded scope arrays in auth.ts or block files (should all use utility functions)
• Each scope has a human-readable description in SCOPE_DESCRIPTIONS within lib/oauth/utils.ts
• No excess scopes that aren't needed by any tool

Step 6: Validate Deployment Availability (if OAuth service)

The deployment UI and setup CLI do not infer OAuth client fields from scopes. They resolve the block's generated oauthServiceId through the application-owned capability catalog.

• The visible integration block has exactly one distinct oauth-input.serviceId
• resolveOAuthClientCapabilityId(serviceId) returns the intended provider capability
• The resolved provider exists in OAUTH_CLIENT_CAPABILITIES
• Every field listed by that capability exists in apps/sim/lib/core/config/env.ts
• Every capability field has the correct text or secret entry in OAUTH_CLIENT_SETUP_FIELDS; no CLI naming heuristic is required
• Shared Google/Microsoft service IDs resolve to their provider capability rather than duplicate entries
• bun run setup integration <capabilityId> is the command emitted by availability; the CLI has only the exhaustive input-mode projection, not a second runtime provider definition
• If the canonical OAuth service declares serviceAccountProviderId, SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID[serviceId] has the same provider ID
• The service-account deploymentRequirement matches how that credential actually works: omitted for an independent path, 'oauth-client' when it needs the OAuth client fields, or 'preview-gated' when controlled by a preview block

Treat a missing capability as critical: runtime availability intentionally throws instead of silently exposing an unusable integration.

Step 7: Validate Pagination Consistency

If any tools support pagination:

• Pagination param names match the API docs (e.g., pagination_token vs next_token vs cursor)
• Different API endpoints that use different pagination param names have separate subBlocks in the block
• Pagination response fields (nextToken, cursor, etc.) are included in tool outputs
• Pagination subBlocks are set to mode: 'advanced'

Step 8: Validate Memory Load Safety

If any tool lists, searches, exports, imports, downloads, uploads, paginates, batches, transforms arrays, or reads file/HTTP bodies, read .agents/skills/memory-load-check/SKILL.md and apply it to the integration.

• List/search tools expose API limits and do not auto-fetch every page into memory
• Transform logic does not build unbounded arrays, maps, sets, or Promise.all fan-outs
• File and HTTP body reads use explicit byte caps or existing stream-limit helpers
• Large result payloads are summarized, paginated, referenced, or capped rather than raw-dumped
• Pagination and download tests cover caps, early stop behavior, or partial-result preservation when relevant

Step 9: Validate Error Handling

• transformResponse checks for error conditions before accessing data
• Error responses include meaningful messages (not just generic "failed")
• HTTP error status codes are handled (check response.ok or status codes)

Step 10: Report and Fix

Report Format

Group findings by severity:

Critical (will cause runtime errors or incorrect behavior):

• Wrong endpoint URL or HTTP method
• Missing required params or wrong required flag
• Incorrect response field mapping (accessing wrong path in response)
• Missing error handling that would cause crashes
• Tool ID mismatch between tool file, registry, and block tools.access
• OAuth scopes missing in auth.ts that tools need
• OAuth integration serviceId missing from the deployment capability catalog
• Capability references an env field absent from the runtime env schema
• Service-account metadata disagrees with the canonical OAuth service configuration
• tools.config.tool returning wrong tool ID for an operation
• Type coercions in tools.config.tool instead of tools.config.params
• Proven model-visible request fields bypass the shared projection or private-provenance boundary
• Opaque model input is downloaded or sent before provenance and workspace-file checks
• A Sim-owned durable sink or internal execution handoff drops encrypted provenance or breaks legacy headerless/NULL data
• A tool substitutes secret plaintext into source, leaks private metadata, or generically sanitizes unrelated third-party results

Warning (follows conventions incorrectly or has usability issues):

• Optional field not set to mode: 'advanced'
• Missing wandConfig on timestamp/complex fields
• Wrong visibility on params (e.g., 'hidden' instead of 'user-or-llm')
• Missing optional: true on nullable outputs
• Opaque type: 'json' without property descriptions
• Missing .trim() on ID fields in request URLs
• Missing ?? null on nullable response fields
• Block condition array missing an operation that uses that field
• Hardcoded scope arrays instead of using getScopesForService() / getCanonicalScopesForProvider()
• Missing scope description in SCOPE_DESCRIPTIONS within lib/oauth/utils.ts

Suggestion (minor improvements):

• Better description text
• Inconsistent naming across tools
• Missing longDescription or docsLink
• Pagination fields that could benefit from wandConfig

Fix All Issues

After reporting, fix every critical and warning issue. Apply suggestions where they don't add unnecessary complexity.

Regenerate Derived Artifacts

Several files are generated from tool and block definitions. Editing a tool or block WITHOUT regenerating them fails CI, so run these before pushing:

bun run tool-metadata:generate       # repo root — apps/sim/tools/generated/*
bun run scripts/generate-docs.ts     # docs .mdx + lib/integrations/integrations.json + docs icons
bun run integration-catalog:check    # registry ↔ committed deployment metadata drift

• tool-metadata:generate — required whenever a tool's outputs, params, or descriptions change. CI enforces this with bun run tool-metadata:check, which fails with "Generated tool metadata is stale". This is the easiest gate to miss, because nothing in the tool file hints that a generated artifact mirrors it.
• generate-docs — required whenever block metadata changes (bgColor, name, description, operations, outputs). Regenerates the integration .mdx, integrations.json, and the docs copy of components/icons.tsx.
• integration-catalog:check — loads the executable block registry, derives visible integration deployment fields, and compares them with the committed catalog. It catches missing/unexpected entries and stale auth/service IDs without loading the executable registry in client code.

Always diff the regen output before committing. These generators rewrite every file they own, so they will also sweep in unrelated drift that accumulated on the base branch — pages losing sections, unrelated icons appearing. Keep only the hunks belonging to the integration under validation and git checkout -- the rest, otherwise an unrelated doc regression rides along in the PR. Verify no page was silently dropped by comparing the directory listing before and after.

If an icon changed, apps/sim/components/icons.tsx is the source of truth and apps/docs/components/icons.tsx is its generated mirror — they must end up byte-identical for that component.

Validation Output

After fixing, confirm:

1. bun run lint passes with no fixes needed
2. TypeScript compiles clean (no type errors) — check the error list is empty for the files you touched; pre-existing unrelated errors in a worktree usually mean workspace packages resolve to the main checkout
3. The integration's tests pass, and any test you added actually fails without its fix (revert it once and watch it go red)
4. Derived artifacts regenerated and their diffs reviewed (see above)
5. bun run integration-catalog:check passes
6. For OAuth or service-account changes, bun test apps/sim/lib/integrations/availability.server.test.ts passes
7. Re-read all modified files to verify fixes are correct
8. Any remaining unknown response schemas were explicitly reported to the user instead of guessed

Checklist Summary

• Read ALL tool files, block, types, index, and registries
• Pulled and read official API documentation
• Validated every tool's ID, params, request, response, outputs, and types against API docs
• Validated block ↔ tool alignment (every tool param has a subBlock, every condition is correct)
• Validated advanced mode on optional/rarely-used fields
• Validated wandConfig on timestamps and complex inputs
• Validated tools.config mapping, tool selector, and type coercions
• Validated block outputs match what tools return, with typed JSON where possible
• Validated OAuth scopes use centralized utilities (getScopesForService, getCanonicalScopesForProvider) — no hardcoded arrays
• Validated scope descriptions exist in SCOPE_DESCRIPTIONS within lib/oauth/utils.ts for all scopes
• Validated OAuth serviceId resolves to the intended OAUTH_CLIENT_CAPABILITIES entry and all capability fields exist in the env schema
• Validated service-account projection and deployment requirement against the canonical OAuth service config
• Regenerated integrations.json when block metadata changed and ran bun run integration-catalog:check
• Validated pagination consistency across tools and block
• Validated memory load safety using .agents/skills/memory-load-check/SKILL.md when tools list/search/download/import/export/batch data
• Validated error handling (error checks, meaningful messages)
• Validated registry entries (tools and block, alphabetical, correct imports)
• Validated model-visible/opaque inputs and Sim-durable/internal-execution provenance at their owning boundaries