coral-create-source-spec

name	coral-create-source-spec
description	Create or update a Coral source spec YAML for a custom HTTP API or local dataset. Use when authoring a standalone source for `coral source add --file`, or when adapting that spec into a Coral repo source under `sources/core` or `sources/community`.

Coral Create Source Spec

Use this skill when the task is to author or repair a Coral source spec.

Goal

Produce a valid, queryable Coral source spec that works with:

• coral source lint <path>
• coral source add --file <path>
• coral source test <name>
• coral sql
• coral.tables and coral.columns
• coral.inputs for source variables and secrets

Default Mode

Default to standalone source authoring for external developers.

That means:

• create a YAML source spec file
• lint it early with coral source lint <path>
• add it to Coral with coral source add --file <path> when you need to exercise it as a source
• validate by querying it
• iterate until the shape is correct

Only switch to Coral repo layout when the user is explicitly editing the Coral repo.

Output Modes

• External authoring:
  • create a standalone source spec such as ./my-source.yaml
  • validate structure with coral source lint ./my-source.yaml
  • load it with coral source add --file ./my-source.yaml when you need to query it through Coral
• Coral repo contribution:
  • write community source specs to sources/community/<name>/manifest.yaml
  • write core source specs to sources/core/<name>/manifest.yaml only when the user is intentionally changing bundled core sources
  • add representative test_queries for a basic smoke/connection check of the source
  • validate with coral source test <name> and repo checks

Workflow

1. Read the provider API docs or inspect the local dataset.
2. Start with one small table and a few columns.
3. Define:
   • source metadata
   • backend
   • base URL or file location
   • auth
   • variables and secrets
   • credential retrieval methods for secrets, including OAuth when the provider supports browser-based setup
   • tables
   • table functions for source-scoped parameterized endpoints
   • filters
   • response extraction
   • pagination
   • typed columns
4. Lint the source spec:
   • coral source lint <path>
5. Validate the source in the right mode:
   • standalone specs: coral source add --file <path> and inspect with coral sql
   • coral source add is non-interactive by default: each input key is read from the matching environment variable. Export required variables and secrets before running, or pass --interactive to be prompted.
   • for OAuth credential methods, run coral source add --interactive --file <path> with no environment value for the target secret so Coral offers the authored credential choices
   • repo sources or already-named sources: coral source test <name>
6. Inspect the exposed shape:
   • inspect coral.tables for visible tables, descriptions, guides, and required filters; keep metadata queries bounded with LIMIT/OFFSET
   • inspect coral.table_functions for source-scoped functions, arguments, result columns, kind, and search limits
   • inspect coral.columns for canonical column metadata, including is_virtual and is_required_filter; filter by one table or page large column sets
   • inspect coral.filters for normalized table filter names, types, modes, required flags, and descriptions
   • inspect coral.inputs to verify variables, secrets, defaults, hints, and required flags
7. Query representative tables with coral sql.
8. If you are relying on coral source test, make sure test_queries gives you a basic smoke/connection check for the source.
9. Refine the spec and repeat.

Authoring Rules

• Start small and expand table coverage incrementally.
• Use the source manifest schema as both inspiration for authoring and validation of structure: https://github.com/withcoral/coral/blob/main/crates/coral-spec/src/schema/source_manifest.schema.json
• Use source variables for non-secret configuration.
• Use source secrets for credentials.
• For OAuth-backed services, model setup with inputs.<TOKEN>.credential.methods[] using type: oauth; keep the runtime auth or request header pointing at the same secret input.
• OAuth credential methods support device-code flow and authorization-code flow. For authorization-code flow, set flow.type: authorization_code, set flow.pkce explicitly to required or disabled, use a loopback http://127.0.0.1 or http://localhost redirect URI, choose redirect_uri_port_mode: random for provider apps that allow variable localhost ports, and choose fixed only when users can register the exact non-zero redirect URI. The CLI also accepts the final loopback redirect URL pasted into the terminal when the browser cannot reach the machine running Coral, so do not reject authorization-code OAuth solely because users may run Coral over SSH, in a VM, or in another split-browser environment. For device-code flow, declare flow.type: device_code, endpoints.device_authorization_url, endpoints.token_url, and a public client ID; omit redirect URI fields and do not declare a client secret.
• OAuth endpoint URLs may template declared kind: variable inputs with {{input.KEY}} for non-secret endpoint components such as tenant IDs or domains. Do not reference secret inputs, filters, function arguments, state, or inline defaults from OAuth endpoint URLs.
• If a provider also supports manually pasted tokens, include a type: source_config fallback after the OAuth method. When the provider's token endpoint requires client authentication with a client secret, prompt for both OAuth client values: declare client.id.input, client.secret.input, and client.secret.transport (basic_auth or request_body).
• Do not add top-level source inputs solely for OAuth client credentials; client.id.input and client.secret.input are collected during OAuth setup.
• For short-lived OAuth access tokens, make sure the OAuth method can obtain refresh tokens when the provider supports them, and document any scopes, consent prompts, or client settings required for refresh-token issuance. If the provider will not issue refresh tokens, call out that users must reconnect when access tokens expire unless the source has another supported long-lived credential path.
• Keep table names stable and SQL-friendly.
• Mark filters as required only when the API truly requires them.
• Use default table functions for parameterized non-retrieval operations, such as scoped child collections, time-range logs, metrics queries, or detail operations that do not map cleanly to a stable table.
• Use kind: search table functions for provider endpoints that accept query text and return ranked candidates.
• Do not model provider search as a table filter. Use mode: contains only for ordinary provider-side substring filters. Provider-ranked retrieval belongs in a kind: search function.
• Include search_limits on every kind: search function and expose stable result identifiers for follow-up detail queries.
• Prefer explicit pagination when the API shape is known.
• Verify pagination with actual row fetches, not only COUNT(*).
• Add or update test_queries when you want coral source test to perform a basic smoke/connection check.

Metadata UX Rules

Use these rules for top-level source metadata so source discovery and setup are consistent.

description

• Start with Query ....
• Make the first sentence capability-first: list the key entities users can query.
• Preferred template:
  • Query <entities> from <Provider> (<Cloud or self-hosted when relevant>).
• Keep description focused on data coverage, not setup steps.
• Do not use vague phrasing such as:
  • REST API v3
  • OpenAPI provider
  • ... and more
• Move auth/setup/permission details to input hints, not description text.

Input hints (inputs.<KEY>.hint)

Each hint should tell the user:

• what value is expected
• how to obtain it
• minimum scope/permission guidance
• one concrete format/example when useful

Specific guidance:

• For URL/base inputs:
  • say what the default means
  • include at least one concrete example
  • include self-hosted guidance when supported
• For secrets:
  • name the exact credential type (API key, PAT, application key, etc.)
  • include format constraints when relevant (for example, token prefixes)
  • include least-privilege scope guidance
• For OAuth methods:
  • use a user-facing label such as Connect with GitHub
  • describe the browser-based setup in description
  • list required OAuth scopes in the secret hint or method description
  • mention whether users need to register a fixed loopback redirect URI or provide their own OAuth client ID/secret
  • for authorization-code flow, note that users can paste the final localhost redirect URL into the terminal if their browser cannot reach Coral's loopback listener directly
• For derived secrets (for example Basic auth blobs):
  • include a short shell example (for example a Base64 command)
• Prefer stable documentation links.
  • Use official docs links and stable settings pages.
  • Avoid brittle click-path instructions as the primary guidance.

Keep hints concise and directly actionable.

Validation Loop

Use this loop during authoring:

# Export any required inputs first (key matches the input `key` in the spec),
# or pass --interactive to be prompted.
coral source lint ./my-source.yaml
coral source add --file ./my-source.yaml
coral source test my_source
coral sql "SELECT schema_name, table_name, description, required_filters FROM coral.tables WHERE schema_name = 'my_source' ORDER BY schema_name, table_name LIMIT 50 OFFSET 0"
coral sql "SELECT function_name, kind, arguments_json, result_columns_json, search_limits_json FROM coral.table_functions WHERE schema_name = 'my_source' ORDER BY function_name LIMIT 50 OFFSET 0"
coral sql "SELECT table_name, filter_name, filter_mode, is_required, data_type, description FROM coral.filters WHERE schema_name = 'my_source' ORDER BY table_name, filter_name LIMIT 100 OFFSET 0"
coral sql "SELECT table_name, column_name, data_type, is_virtual, is_required_filter, filter_mode, description FROM coral.columns WHERE schema_name = 'my_source' ORDER BY table_name, ordinal_position LIMIT 100 OFFSET 0"
coral sql "SELECT key, kind, value, default_value, hint, required, is_set FROM coral.inputs WHERE schema_name = 'my_source' ORDER BY key"

For repo sources or already-named sources, add test_queries for a basic smoke/connection check and run:

coral source test my_source

Then run targeted table queries until the source behaves correctly.

HTTP Sources

For HTTP-backed sources:

• define backend: http
• define base_url
• define auth headers or other runtime auth fields
• define credential.methods on secret inputs when setup should offer OAuth or another retrieval choice
• define request path, query, and body only where needed
• define source-scoped table functions for provider-native operations that require invocation arguments
• define response rows_path
• define pagination explicitly when the provider pattern is known
• define typed columns
• add test_queries once you know which simple query or queries should confirm the source basically works

Read references/http-source-checklist.md when you need table-shape and pagination guidance.

If your HTTP source uses an Authorization header with a prefix (e.g. Authorization: Bearer <token>), use a secret input for the raw token and define the header with from: bearer:

inputs:
  FOOBAR_API_TOKEN:
    kind: secret
    hint: Bearer token for the Foobar API.
auth:
  type: HeaderAuth
  headers:
    - name: Authorization
      from: bearer
      key: FOOBAR_API_TOKEN

For an OAuth-backed HTTP source, add the retrieval method to that same secret input:

inputs:
  FOOBAR_API_TOKEN:
    kind: secret
    hint: Access token for the Foobar API. Use OAuth setup or paste a token with read access.
    credential:
      methods:
        - type: oauth
          label: Connect with Foobar
          description: Open a browser and authorize Coral to read Foobar data.
          oauth:
            flow:
              type: authorization_code
              pkce: required
            redirect_uri: http://127.0.0.1:0/oauth/callback
            redirect_uri_port_mode: random
            endpoints:
              authorization_url: https://foobar.example.com/oauth/authorize
              token_url: https://foobar.example.com/oauth/token
            client:
              id:
                input: FOOBAR_OAUTH_CLIENT_ID
            scopes:
              scope:
                delimiter: space
                values:
                  - read
        - type: source_config
          label: Paste token
auth:
  type: HeaderAuth
  headers:
    - name: Authorization
      from: bearer
      key: FOOBAR_API_TOKEN

When a provider accepts either a full pasted API-key header or an OAuth access token, declare both credential inputs as optional secrets, then use from: one_of and put the complete header value first, followed by a from: bearer OAuth fallback:

inputs:
  FOOBAR_API_KEY:
    kind: secret
    required: false
  FOOBAR_OAUTH_ACCESS_TOKEN:
    kind: secret
    required: false
auth:
  type: HeaderAuth
  headers:
    - name: Authorization
      from: one_of
      values:
        - from: input
          key: FOOBAR_API_KEY
        - from: bearer
          key: FOOBAR_OAUTH_ACCESS_TOKEN

Local Data Sources

For local file-backed sources:

• define the file backend
• define the source location
• define file selection patterns if applicable
• define typed columns

Deliverable

Report:

• source spec path
• lint / add / test commands used
• validation commands run
• assumptions made
• blocked or unverified endpoints