extract

name	extract
description	Extract both schema and data from a JDBC source

Extract Skill

Combines schema extraction and data extraction in a single command. First extracts the database schema metadata into Starlake YAML files, then extracts the actual data into files. This is a convenience command that runs extract-schema followed by extract-data.

Usage

starlake extract [options]

Options

Combines all options from extract-schema and extract-data.

Schema Extraction Options

• --config <value>: Database tables & connection info
• --outputDir <value>: Where to output YML files
• --tables <value>: Database tables to extract
• --connectionRef <value>: JDBC connection reference
• --all: Extract all schemas and tables
• --external: Output YML files to the external folder
• --parallelism <value>: Parallelism level
• --snakecase: Apply snake_case to column names

Data Extraction Options

• --limit <value>: Limit number of records
• --numPartitions <value>: Partition parallelism
• --ignoreExtractionFailure: Continue on extraction failure
• --clean: Clean target files before extraction
• --incremental: Export only new data since last extraction
• --includeSchemas <value>: Domains to include
• --excludeSchemas <value>: Domains to exclude
• --includeTables <value>: Tables to include
• --excludeTables <value>: Tables to exclude
• --reportFormat <value>: Report output format: console, json, or html

Configuration Context

Extract commands use a configuration file (metadata/extract/{name}.sl.yml) to define which schemas and tables to extract:

# metadata/extract/externals.sl.yml
version: 1
extract:
  connectionRef: "duckdb"
  jdbcSchemas:
    - schema: "starbake"
      tables:
        - name: "*"              # "*" to extract all tables
      tableTypes:
        - "TABLE"

Advanced Extract Configuration

# metadata/extract/source_db.sl.yml
version: 1
extract:
  connectionRef: "source_postgres"
  jdbcSchemas:
    - schema: "sales"
      tableTypes:
        - "TABLE"
        - "VIEW"
      tables:
        - name: "orders"
          fullExport: false          # Incremental extraction
          partitionColumn: "id"      # Column for parallel extraction
          numPartitions: 4           # Parallelism level
          timestamp: "updated_at"    # Incremental tracking column
          fetchSize: 1000            # JDBC fetch size
        - name: "customers"
          fullExport: true

Connection Configuration

The connection referenced in the extract config must be defined in application.sl.yml:

# metadata/application.sl.yml
version: 1
application:
  connections:
    source_postgres:
      type: jdbc
      options:
        url: "jdbc:postgresql://{{PG_HOST}}:{{PG_PORT}}/{{PG_DB}}"
        driver: "org.postgresql.Driver"
        user: "{{DATABASE_USER}}"
        password: "{{DATABASE_PASSWORD}}"

REST API Extract Configuration

Extract schemas and data from REST API endpoints. See dedicated skills for full details:

• extract-rest-schema: infer schemas from API responses
• extract-rest-data: extract data to CSV with pagination, auth, rate limiting

# metadata/extract/my-rest-api.sl.yml
version: 1
extract:
  restAPI:
    baseUrl: "https://api.example.com/v2"
    auth:
      type: bearer
      token: "{{API_TOKEN}}"
    rateLimit:
      requestsPerSecond: 10
    defaults:
      pagination:
        type: offset
        limitParam: "limit"
        offsetParam: "offset"
        pageSize: 100
    endpoints:
      - path: "/customers"
        as: "customer"
        domain: "crm"
        responsePath: "$.data"
        incrementalField: "updated_at"

Supported auth types: bearer, api_key, basic, oauth2_client_credentials. Supported pagination: offset, cursor, link_header, page_number.

OpenAPI Extract Configuration

Extract schemas from OpenAPI/Swagger specifications:

# metadata/extract/api.sl.yml
version: 1
extract:
  openAPI:
    basePath: /api/v2

    domains:
      - name: customers_api

        # Schema filtering (regex)
        schemas:
          exclude:
            - "Model\\.Common\\.Id"
            - "Internal\\..*"

        # Route selection
        routes:
          - paths:
              include:
                - "/users"
                - "/orders"
                - "/products"

Freshness Monitoring

Track data freshness with timestamp columns after extraction:

# Check freshness for specific tables
starlake freshness --tables dataset1.table1,dataset2.table2 --persist true

Monitoring table: SL_LAST_EXPORT in audit schema.

Examples

Extract Schema and Data

starlake extract --config externals --outputDir metadata/load

Extract with Incremental Mode

starlake extract --config source_db --outputDir /tmp/output --incremental

Extract Specific Tables

starlake extract --config source_db --tables sales.orders,sales.customers

Related Skills

• extract-schema - Extract schema from JDBC databases
• extract-data - Extract data from JDBC databases
• extract-rest-schema - Extract schema from REST API endpoints
• extract-rest-data - Extract data from REST API endpoints
• extract-script - Generate extraction scripts from templates
• freshness - Check data freshness
• load - Load extracted data into the warehouse