| name | worker-k8s-external |
| description | Deploy the Data Exchange Worker on an external Kubernetes cluster — image selection, Kubernetes Secret creation, Deployment manifest generation, and verification. Triggers: kubernetes worker, k8s worker, kubectl worker, external cluster worker, helm worker. |
Worker Kubernetes Setup — External Cluster
Deploy the Data Exchange Worker as a Kubernetes Deployment on a customer-managed cluster. Unlike SPCS, there is no injected Snowflake token — both source database and Snowflake credentials are supplied explicitly via Kubernetes Secrets.
Step 1 — Image Version
The worker image version is pinned to your scai CLI release — do not use SHOW IMAGES to pick the latest tag, as it may be incompatible with this version of the CLI.
Get the pinned version by running:
scai data worker setup --compute-pool <any-pool>
The CLI logs: Using pinned Data Exchange Worker image: /snowflake/images/snowflake_images/data-exchange-worker:<version>
Record that exact version tag (e.g., 1.11.1). Use it as <version> in all subsequent steps. You can cancel the setup command once you have the version — this step is only needed to read the tag from CLI output. Upgrading the version later requires re-running this skill from the beginning.
Note: The image tag is a version number (e.g., 1.12.0), not a platform identifier. The source type is controlled by the DATA_SOURCE_TYPE environment variable. Read it from the project's scai source connection:
| Source system | DATA_SOURCE_TYPE |
|---|
| SQL Server | sqlserver |
| Redshift | redshift |
| Oracle | oracle |
| Teradata | teradata |
| PostgreSQL | postgresql |
Step 2 — Affinity Label
In all steps below, substitute every <placeholder> token with its actual value — never output a literal <placeholder> string. Key substitutions:
<label> — the affinity label confirmed in this step<version> — the image version from Step 1<source_type>, <host>, <port>, <database> — from the project's scai source connection<replica_count> — the count the user provides in Step 3<source_username>, <source_password> — source database credentials from the project's scai source connection; do not ask the user to type these in the chat — instruct them to fill them directly into the command<snowflake_account>, <snowflake_user>, <snowflake_password>, <warehouse> — Snowflake credentials collected from the user in Step 4; do not ask the user to say passwords in the chat — instruct them to fill them directly into the command
Derive the affinity label from the project directory name: lowercase, spaces replaced with hyphens, special characters stripped.
Example: project at /migrations/acme corp → acme-corp
Tell the user:
Your affinity label is <label>. All worker pods will use this label, and you will set the same label in your migration workflow config. Confirm or provide a different label.
Wait for the user to confirm or override.
Step 3 — Worker Count
Tell the user:
How many worker pods should you run?
- Start with 2–3 replicas for most migrations.
- Add more if the source system can sustain higher read throughput and your network has the bandwidth.
- Monitor source system CPU and I/O before scaling up — too many workers can disrupt production workloads.
You can scale the deployment at any time with:
kubectl scale deployment data-exchange-worker --replicas=<n>
Ask for the desired replica count.
Wait for the user to provide the replica count before continuing to Step 4.
Step 4 — Kubernetes Secret
Unlike SPCS, there is no injected Snowflake token. Credentials for both the source database and Snowflake must be provided.
Pre-fill source database values from the project's scai source connection. Collect the following Snowflake values from the user:
SNOWFLAKE_ACCOUNT — account identifier (e.g., myorg-myaccount)SNOWFLAKE_USER — Snowflake usernameSNOWFLAKE_PASSWORD — Snowflake passwordSNOWFLAKE_WAREHOUSE — warehouse name (already configured in parent skill Question 2)
Show the following command and instruct the user to fill in their actual credentials before running it:
kubectl create secret generic data-exchange-worker-secrets \
--from-literal=DATA_SOURCE_USERNAME='<source_username>' \
--from-literal=DATA_SOURCE_PASSWORD='<source_password>' \
--from-literal=SNOWFLAKE_ACCOUNT='<snowflake_account>' \
--from-literal=SNOWFLAKE_USER='<snowflake_user>' \
--from-literal=SNOWFLAKE_PASSWORD='<snowflake_password>' \
--from-literal=SNOWFLAKE_WAREHOUSE='<warehouse>'
Wait for the user to confirm the secret has been created before continuing.
Next, create an image pull secret so your cluster can pull from the Snowflake Image Repository. Ask the user for their Snowflake account registry hostname — it follows the pattern <orgname>-<accountname>.registry.snowflakecomputing.com. If they are unsure, they can run SHOW IMAGE REPOSITORIES IN SCHEMA snowflake.images; to find the repository_url.
Show:
docker login <registry_hostname>
kubectl create secret generic snowflake-registry-pull \
--from-file=.dockerconfigjson="${HOME}/.docker/config.json" \
--type=kubernetes.io/dockerconfigjson
Wait for the user to confirm the pull secret has been created before continuing.
Step 5 — Generate Deployment YAML
Write the following manifest to .scai/worker-deployment.yaml in the project directory, substituting all non-credential placeholder values:
apiVersion: apps/v1
kind: Deployment
metadata:
name: data-exchange-worker
labels:
app: data-exchange-worker
spec:
replicas: <replica_count>
selector:
matchLabels:
app: data-exchange-worker
template:
metadata:
labels:
app: data-exchange-worker
spec:
containers:
- name: agent
image: snowflake/images/snowflake_images/data-exchange-worker:<version>
env:
- name: DATA_SOURCE_TYPE
value: "<source_type>"
- name: DATA_SOURCE_HOST
value: "<host>"
- name: DATA_SOURCE_PORT
value: "<port>"
- name: DATA_SOURCE_DATABASE
value: "<database>"
- name: MAX_PARALLEL_TASKS
Show the generated file to the user.
Note: SNOWFLAKE_CONNECTION_NAME: "MY_SNOWFLAKE_CONNECTION" is the fixed connection name expected by the Data Exchange Worker in external Kubernetes mode. Do not change this value.
Oracle/Teradata: Note that pods must have outbound network access (port 443) to download the database driver on startup. No manifest change is required, but the cluster's network policy must allow this egress.
Note: MAX_PARALLEL_TASKS defaults to 4. Reduce this value if the source system cannot handle that many concurrent connections.
Wait for the user to confirm the manifest before applying.
Step 6 — Apply and Verify
Apply the manifest:
kubectl apply -f .scai/worker-deployment.yaml
Check pod status:
kubectl get pods -l app=data-exchange-worker
Expected: all pods reach Running state. Check logs to confirm the worker is connected and polling:
kubectl logs -l app=data-exchange-worker --tail=50
Wait for the user to confirm all pods are running and polling for tasks before continuing.
Step 7 — Remind
Tell the user:
Important: When you configure your migration workflow, set the affinity to <label>. Without this, the orchestrator will not route tasks to your workers.
To check pod status at any time:
kubectl get pods -l app=data-exchange-worker
kubectl logs -l app=data-exchange-worker --tail=50
Done
The worker deployment is running on the external cluster and all pods are polling for tasks. Return control to the parent skill, which runs Level 1 Data Doctor before proceeding.
