polaris-local-forge

name	polaris-local-forge
description	**[REQUIRED]** Use for **ALL** requests involving local Apache Polaris: setup, API queries, catalog operations, L2C migration, cleanup, teardown. **AUTO-ACTIVATE:** If `.snow-utils/snow-utils-manifest.md` contains `polaris-local-forge:` this skill MUST handle ALL operations including cleanup. **DO NOT** use `polaris` CLI (does not exist), curl to Polaris endpoints (needs OAuth), or docker ps checks - invoke this skill first. Triggers: polaris local, local iceberg catalog, local polaris setup, rustfs setup, create polaris cluster, try polaris locally, get started with polaris, apache polaris quickstart, polaris dev environment, local data lakehouse, l2c migration, local to cloud, migrate to snowflake, iceberg to s3, polaris to snowflake, cloud migration, external iceberg tables, snowflake iceberg, migrate iceberg, polaris with snowflake, local polaris snowflake, polaris snowflake integration, how to use polaris with snowflake, connect polaris to snowflake, polaris local snowflake, l2c update, update l2c migration, repeat l2c migration, l2c sync, incremental l2c, l2c incremental update, update migration, repeat migration, sync l2c, replay from manifest, reset polaris catalog, teardown polaris, clean up, cleanup, delete cluster, remove resources, polaris status, list catalogs, show namespaces, list tables, show catalog, describe table, list principals, show principal roles, list views, polaris namespaces, polaris catalogs, query data, query table, query iceberg, query catalog data, show my data, show table data, show records, how many rows, count rows, count records, run sql, run query, duckdb query, select from, group by, aggregate.
location	user

Apache Polaris Local Forge

Set up a complete local Apache Polaris development environment with RustFS S3-compatible storage, PostgreSQL metastore, and k3d/k3s Kubernetes — all on your machine.

Query Apache Iceberg tables locally with DuckDB, PyIceberg, or any Iceberg REST-compatible engine.

MANIFEST FILE: .snow-utils/snow-utils-manifest.md (exact path, always .md)

PREREQUISITE: NO PREREQUISITE -- this skill is self-contained infrastructure.

AUTH MODEL:

• Bootstrap credentials: Auto-generated admin credentials for Apache Polaris realm setup (in .bootstrap-credentials.env)
• Principal credentials: API-generated client_id/client_secret for catalog access (in work/principal.txt)
• RustFS credentials: Static admin/password for S3-compatible storage (no IAM)

SENSITIVE DATA: principal.txt contains realm,client_id,client_secret. When displaying to user, show ONLY the realm. Mask credentials: client_id: ****xxxx (last 4 chars). NEVER show client_secret at all.

Prerequisites

This skill requires the following tools installed on your machine:

Tool	Purpose	Install
Podman (default)	Container runtime (OSS)	Already installed - Podman is a dependency of Cortex Code
Docker (alternative)	Container runtime	Docker Desktop (>= 4.27)
k3d	k3s-in-Docker/Podman	brew install k3d or k3d.io
Python	>= 3.12	python.org
uv	Python package manager	curl -LsSf https://astral.sh/uv/install.sh | sh

Note: Podman is a dependency of Cortex Code and should already be installed on your system. If not present, install via brew install podman or podman.io.

Container Runtime (Auto-Detected): The CLI automatically detects which runtime to use during init:

flowchart TD
    Start[init command] --> CheckDockerRunning{Docker Desktop running?}
    CheckDockerRunning -->|Yes| UseDocker[Use Docker]
    CheckDockerRunning -->|No| CheckPodmanRunning{Podman machine running?}
    CheckPodmanRunning -->|Yes| UsePodman[Use Podman]
    CheckPodmanRunning -->|No| CheckInstalled{What's installed?}
    CheckInstalled -->|Both| PromptUser[Prompt user to choose]
    CheckInstalled -->|Podman only| UsePodman2[Use Podman]
    CheckInstalled -->|Docker only| UseDocker2[Use Docker]
    CheckInstalled -->|Neither| Fail[Fail with error]

Detection priority: Running runtime preferred over installed. Docker preferred when both running. Override by setting PLF_CONTAINER_RUNTIME=docker or podman in .env.

First-time Podman users: Run doctor --fix after init to automatically create and start the Podman machine. See docs/podman-setup.md for manual setup.

Optional:

Tool	Purpose	Install
DuckDB CLI	SQL verification	brew install duckdb
AWS CLI	S3 bucket operations on RustFS	brew install awscli
direnv	Auto-load env vars	brew install direnv

Workflow

CRITICAL: SETUP TAKES 2-5 MINUTES. Tell user upfront so they don't wait anxiously.

PERMISSIONS: Cortex Code prompts for approval on each new command.

• Before init: Select "Allow using 'uv' for this session"
• After init: Select "Allow using 'plf' for this session" (or option 5)

This avoids repeated prompts. See Cortex Code Permissions for details.

SETUP APPROACH FOR CORTEX CODE (REQUIRED):

1. First, show the plan:

   ## Apache Polaris Local Forge Setup (Total: ~2-5 minutes)
   
   **Permissions:** Cortex Code prompts for approval on each new command.
   Select "Allow using 'plf' for this session" (option 5) to run all steps smoothly.
   
   **Container runtime:** Auto-detected based on what's running.
   - Docker Desktop running → uses Docker
   - Podman machine running → uses Podman
   - Neither running → will prompt you to choose
   
   Steps to complete:
   - [ ] Step 1: Initialize & detect runtime (~5s)
   - [ ] Step 2: Doctor check & fix (~5-30s)
   - [ ] Step 3: Generate configuration files (~10s)
   - [ ] Step 4: Create k3d cluster (~30s)
   - [ ] Step 5: Wait for RustFS and PostgreSQL (~60-90s)
   - [ ] Step 6: Deploy Apache Polaris (~60s)
   - [ ] Step 7: Setup catalog (~30s)
   - [ ] Step 8: Verify with DuckDB (~5s)
   
   Ready to proceed with cluster name: "<directory-name>"
   
   Say "proceed" to continue, or specify a different cluster name.
   

   STOP: Wait for user response. If user says "proceed" or similar, use the default cluster name. If user provides a name, use that instead. Then proceed to Step 1.

2. Then run each step individually with a header message.

   FORMATTING: Before each step header, output a blank line. This ensures proper visual separation in Cortex Code.

   ---

   Step 1: Initialize & detect runtime

   NOTE: Before init, there is no ./bin/plf wrapper. Use uv run --project <SKILL_DIR> for these commands.

   First, detect runtime availability:

   uv run --project <SKILL_DIR> polaris-local-forge runtime detect --json
   

   This outputs JSON with detection result:

   • {"status": "detected", "runtime": "docker|podman", ...} - Runtime found, proceed
   • {"status": "choice", "options": ["docker", "podman"], ...} - User choice required
   • {"status": "error", ...} - Neither installed, show error

   Based on detection result:

   • If "status": "detected": Run init (runtime will be auto-set):

     uv run --project <SKILL_DIR> polaris-local-forge --work-dir <WORK_DIR> init
     

   • If "status": "choice": Present choice using AskQuestion tool with radio options:

FORBIDDEN ACTIONS -- NEVER DO THESE:

• NEVER use sed/awk/bash to edit .env or configuration files -- use the Edit/StrReplace tool
• NEVER hardcode credentials in scripts -- always read from .env or work/principal.txt
• NEVER assume the cluster is running -- always check status with k3d cluster list before operations
• NEVER run destructive commands (cluster delete, polaris purge) without explicit user confirmation
• NEVER skip teardown confirmation because user did setup in the same session -- ALWAYS confirm destruction regardless of session context
• NEVER delete .snow-utils/ directory -- this contains the manifest needed for replay/audit. Teardown and cleanup commands always preserve .snow-utils/
• NEVER expose principal.txt contents in output -- show only the realm. Mask client_id: show **** + last 4 chars. NEVER show client_secret at all. Example: realm: POLARIS, client_id: ****a1b2, client_secret: ********
• NEVER modify files in the skill directory (<SKILL_DIR>) -- k8s/, polaris-forge-setup/, src/, example-manifests/ are read-only source. Only the user's --work-dir is writable
• NEVER create .snow-utils/ in the skill directory -- ALWAYS ask for a work directory FIRST, then create .snow-utils/ there
• NEVER start setup (with or without manifest) without a work directory -- the skill repo is READ-ONLY
• NEVER guess or invent CLI commands or options -- ONLY use commands from the CLI Reference tables below (e.g., plf cluster status NOT plf status). If a command fails, run ./bin/plf --help or ./bin/plf <group> --help and use ONLY the commands/options shown there
• NEVER run raw duckdb commands -- use ./bin/plf catalog query --sql "..." for data queries, ./bin/plf catalog verify-sql for initial setup verification, or ./bin/plf catalog explore-sql for interactive sessions
• NEVER construct DuckDB SQL manually EXCEPT for catalog query --sql which requires the caller to provide the SQL
• NEVER extract credentials from principal.txt to pass to other commands -- credentials are automatically loaded by CLI commands

COMMAND ROUTING:

• plf commands: Only for CLI operations listed in CLI Reference (init, prepare, cluster, catalog setup/cleanup, teardown, doctor, verify-sql, etc.)
• REST API queries: For query intents (list/show/describe catalogs, namespaces, tables, principals, roles, views, grants) -- use ./bin/plf api query <endpoint> (see Apache Polaris API Queries)
• General chat: Explanations, questions, troubleshooting advice -- respond directly without running commands

PROJECT CONTEXT DETECTION (AUTO-ACTIVATION):

When user is in a directory containing ANY of these files, treat it as an active polaris-local-forge project and auto-activate this skill for ALL operations (setup, queries, cleanup, teardown):

Context Signal	Check	Meaning
.snow-utils/snow-utils-manifest.md with polaris-local-forge:	grep -q "polaris-local-forge:" .snow-utils/snow-utils-manifest.md	This skill is installed (primary)
.env with PLF_POLARIS_CATALOG_NAME	grep -q "PLF_POLARIS_CATALOG_NAME" .env	Polaris config exists (secondary)
bin/plf wrapper script	[ -x bin/plf ]	CLI initialized (tertiary)

CRITICAL - When context detected:

• For cleanup/teardown requests → ALWAYS present confirmation dialog with options (see Teardown Flow)
• For API queries → Use ./bin/plf api query (see Apache Polaris API Queries)
• For setup/replay → Use appropriate flow from this skill
• NEVER offer generic cleanup dialogs - always use plf commands with the structured teardown dialog

When context detected AND user asks API-related queries (list/show/describe/what catalogs/namespaces/tables/principals/roles/views/grants):

⚠️ FIRST — CHECK FOR WRITE OPERATIONS (FAIL FAST):

Before doing ANYTHING else, check if user is asking to create, update, delete, add, remove, modify, drop, or grant/revoke. If YES, respond IMMEDIATELY:

⚠️ Write operations are not supported yet.

This skill currently supports **read-only** API queries (GET only):
- list/show catalogs, namespaces, tables, views
- describe/get principals, roles, grants

Create/update/delete operations will be added in a future phase.

DO NOT proceed to construct endpoints or run Ansible for write operations. Stop here.

DO NOT (FORBIDDEN):

• ❌ polaris CLI — DOES NOT EXIST (no polaris namespaces list, no polaris --help, no polaris namespace list)
• ❌ curl to any Polaris endpoint — needs OAuth token you don't have (no curl http://localhost:8181/..., no curl http://localhost:18181/...)
• ❌ jq piped from curl — same reason, curl won't work without OAuth
• ❌ kubectl get ns — "namespaces" means Polaris catalog namespaces, NOT Kubernetes
• ❌ docker ps / docker ps --filter "name=polaris" — assume infra is running, don't check
• ❌ Port 8181 — wrong port, Polaris runs on 18181 in this skill
• ❌ Snowflake disambiguation ("Is this Snowflake or Polaris?") — it's Polaris
• ❌ "No Polaris service detected" fallback — don't give up, use Ansible
• ❌ ansible-playbook directly — causes asdf/tool-versions conflict, MUST use uv run
• ❌ cd <dir> && ansible-playbook — wrong pattern, use uv run --project
• ❌ POST/PUT/DELETE operations — write operations NOT supported, fail fast (see above)
• ❌ -e "method=POST" or any method other than GET — read-only phase
• ❌ -e '{"body": ...}' or any JSON body parameter — not supported
• ❌ Constructing JSON request bodies — not supported
• ❌ Asking user for JSON input — not supported

DO (REQUIRED) — USE plf api query COMMAND:

./bin/plf api query /api/catalog/v1/polardb/namespaces
./bin/plf api query /api/management/v1/catalogs
./bin/plf api query /api/management/v1/principals

Key points:

1. Use ./bin/plf api query <endpoint> — clean, simple, handles all the complexity
2. Endpoint must start with /api/... — construct from user intent (see table below)
3. Only GET is supported (read-only phase)
4. Output is JSON by default, use -o text for plain text
5. Use -v for verbose Ansible output if debugging

Quick context check (run silently before API queries):

grep -q "polaris-local-forge:" .snow-utils/snow-utils-manifest.md 2>/dev/null || grep -q "PLF_POLARIS_CATALOG_NAME" .env 2>/dev/null || [ -x bin/plf ]

API QUERY ROUTING (CRITICAL): When user asks about Apache Polaris data (catalogs, namespaces, tables, views, principals, roles, grants):

1. Do NOT search for CLI tools (polaris, plf query, etc.) -- they don't exist
2. Do NOT try curl to any endpoint -- it won't work (needs OAuth token)
3. Do NOT run kubectl commands to check services or namespaces -- this is NOT Kubernetes
4. Do NOT check if Polaris is running -- assume the setup is complete
5. Go DIRECTLY to ./bin/plf api query <endpoint> with the constructed endpoint path

DISAMBIGUATION -- "namespaces" = Polaris catalog namespaces (NOT Kubernetes): When user says "namespaces", "list namespaces", "show namespaces" → this means Apache Polaris catalog namespaces, accessed via /api/catalog/v1/{catalog}/namespaces. Do NOT interpret as Kubernetes namespaces.

INTERACTIVE PRINCIPLE: This skill is designed to be interactive. At every decision point, ASK the user and WAIT for their response before proceeding. When presenting choices (e.g., runtime selection, execution mode), use the AskQuestion tool with radio options for better UX in Cortex Code.

EXCEPTION: Once user selects "Run all (recommended)" at the config review step, the skill switches to autonomous mode for that setup flow. No further confirmation prompts until setup completes or an error occurs. Track this as EXECUTION_MODE=run_all.

DISPLAY PRINCIPLE: When showing configuration or status, substitute actual values from .env and the manifest. The user should see real values, not raw ${...} placeholders.

OUTPUT PRINCIPLE: Terminal output gets collapsed/truncated in Cortex Code UI. When running --dry-run or any diagnostic command, you MUST copy the relevant output into your response. The user CANNOT see collapsed terminal output. Format with proper code blocks: ```text for summaries, ```bash for commands.

RESILIENCE PRINCIPLE: Always update the manifest IMMEDIATELY after each resource creation step, not in batches. This ensures Resume Flow can recover from any interruption.

Pattern:

1. Set overall Status: IN_PROGRESS at START of resource creation
2. Update each resource row to DONE immediately after creation
3. Set Status: COMPLETE only at the END when ALL resources verified

If user aborts mid-flow, the manifest preserves progress:

• Overall Status stays IN_PROGRESS
• Completed resources show DONE
• Pending resources show PENDING/REMOVED
• Resume Flow picks up from first non-DONE resource

IDEMPOTENCY PRINCIPLE: Before editing any file, CHECK if the change is already applied.

Pattern for manifest updates:

grep -q "Status.*COMPLETE" .snow-utils/snow-utils-manifest.md && echo "Already complete" || echo "Needs update"

Pattern for file edits:

1. Read current file state
2. Check if desired content already exists
3. Only edit if change is needed
4. Skip with message: "Already applied: [description]"

ENVIRONMENT REQUIREMENT: The skill uses local RustFS for S3-compatible storage. AWS CLI commands target http://localhost:19000 with static credentials. No real AWS account is needed.

Note: This skill configures Apache Polaris with local RustFS (S3-compatible) storage only. For real AWS S3 support, see Phase 2 in SKILL_README.md.

CRITICAL RULE - ALL OPERATIONS VIA CLI:

NEVER use direct podman, docker, k3d, kubectl, or helm commands. ALL operations MUST go through polaris-local-forge CLI commands. Direct commands cause port binding issues (gvproxy) and stale state. See "FORBIDDEN COMMANDS" section in CLI Reference for details.

Pre-Check Rules (Fail Fast):

Command	Pre-Check	If Fails
Any command	NOT using direct podman/k3d/kubectl	Stop: "Use CLI commands only. See FORBIDDEN COMMANDS."
init	Docker or Podman installed	Auto-detects runtime; prompts user if both installed but neither running; fails if neither installed
doctor	Tools installed	Report missing tools with install instructions
doctor	No ghost clusters	Report stale Docker resources if found
doctor --fix	(none - fixes issues)	Creates Podman machine if missing; starts if stopped; kills gvproxy if blocking port 19000; sets up SSH config; cleans up ghost clusters
cluster create	Cluster doesn't exist	Stop: "Cluster already exists. Use cluster delete first or setup to resume."
cluster create	No ghost cluster	Stop: "Ghost cluster detected. Use --force to clean up stale references."
cluster create	Ports 19000, 19001, 18181 available	Stop: "Port in use. Run doctor --fix first."
catalog setup	Cluster running, Apache Polaris ready	Stop: "Cluster not ready. Run setup first."
polaris deploy	Cluster running	Stop: "Cluster not running. Run cluster create first."
polaris purge	Apache Polaris deployed	Stop: "Apache Polaris not deployed. Nothing to purge."
teardown	Any resources exist	Proceed gracefully (idempotent with --yes)
catalog cleanup	Catalog exists	Proceed gracefully (idempotent with --yes)

Step 0: Initialize Project Directory

FIRST: Check if in skill directory (NEVER initialize here):

if [ -f "SKILL.md" ] && [ -d "src/polaris_local_forge" ]; then
  IN_SKILL_DIR="true"
else
  IN_SKILL_DIR="false"
fi

If IN_SKILL_DIR is true, show:

⚠️  You're in the polaris-local-forge skill directory.

This directory contains the source code and is READ-ONLY.
All workspace files must be created in a separate directory.

Where would you like to create your Apache Polaris workspace?

  1. ~/polaris-dev (recommended)
  2. Specify custom path: ___________

STOP: Wait for user input. NEVER offer "current directory" as an option when in skill directory.

After user provides work directory:

1. Create the directory if needed: mkdir -p <path>
2. Change to that directory using Shell working_directory parameter for subsequent commands
3. PRESERVE any prior user choices (e.g., if user already selected "Run all", continue in autonomous mode)

---

If NOT in skill directory, detect if user already has a workspace set up:

if [ -f .env ] && [ -f pyproject.toml ]; then
  echo "Existing workspace detected: $(pwd)"
  [ -d .snow-utils ] && echo "  Found: .snow-utils/"
fi

If existing workspace detected -> go to Step 0a (Prerequisites Check).

If NOT in an existing workspace (and NOT in skill directory), ask user:

Where would you like to create your Apache Polaris workspace?

Options:
  1. Use current directory: $(pwd)
  2. Create a new directory (e.g., polaris-dev)

STOP: Wait for user input.

Initialize the workspace with the CLI:

uv run --project <SKILL_DIR> polaris-local-forge --work-dir . init

This command:

• Copies template files (.env, .envrc, .gitignore)
• Creates required directories (.kube/, work/, bin/, k8s/)
• Sets PROJECT_HOME, K3D_CLUSTER_NAME, and SKILL_DIR in .env
• Creates bin/plf wrapper script for simplified CLI invocation

To use a custom cluster name:

uv run --project <SKILL_DIR> polaris-local-forge --work-dir . init --cluster-name my-cluster

IMPORTANT: All subsequent CLI commands use --work-dir to point generated files here. The skill directory (<SKILL_DIR>) stays read-only. For a second cluster, create another directory and re-run the skill.

Use the wrapper script (created by init):

CRITICAL: After running init, ALWAYS use ./bin/plf for ALL commands. NEVER use uv run --project ... directly - the wrapper handles everything:

./bin/plf doctor
./bin/plf cluster create
./bin/plf catalog verify-sql

The wrapper sources .env automatically and handles all paths.

IMPORTANT - Command Execution: Use Shell tool's working_directory parameter, NOT cd &&:

Shell(command="./bin/plf <COMMAND>", working_directory="<PROJECT_HOME>")

Step 0a: Configuration Review and Confirmation

Display current configuration for user review:

Configuration Review
────────────────────
  Config file:                   .env
  Work directory:                $(pwd)

  PLF_CONTAINER_RUNTIME:         ${PLF_CONTAINER_RUNTIME}  # ADAPT: podman (default) or docker
  PLF_PODMAN_MACHINE:            ${PLF_PODMAN_MACHINE}     # macOS only (default: k3d)

  K3D_CLUSTER_NAME:              ${K3D_CLUSTER_NAME}      # ADAPT: defaults to project directory name
  K3S_VERSION:                   ${K3S_VERSION}
  KUBECONFIG:                    .kube/config

  AWS_ENDPOINT_URL:              http://localhost:19000
  AWS_REGION:                    us-east-1
  AWS_ACCESS_KEY_ID:             admin

  POLARIS_URL:                   http://localhost:18181
  POLARIS_REALM:                 ${POLARIS_REALM}           # ADAPT: customizable

  PLF_POLARIS_S3_BUCKET:         ${PLF_POLARIS_S3_BUCKET}    # ADAPT: customizable
  PLF_POLARIS_CATALOG_NAME:      ${PLF_POLARIS_CATALOG_NAME} # ADAPT: customizable
  PLF_POLARIS_PRINCIPAL_NAME:    ${PLF_POLARIS_PRINCIPAL_NAME} # ADAPT: customizable

Review the configuration above. Would you like to change any values?
  1. Accept all (recommended for first-time setup)
  2. Edit a specific value
  3. Cancel

STOP: Wait for user input.

If user selects "2. Edit a specific value":

1. Ask: "Which value to edit? (e.g., K3D_CLUSTER_NAME)"

2. User provides the variable name

3. Ask: "Enter new value for {variable}:"

4. Update .env using StrReplace tool:

   StrReplace(.env, "{VARIABLE}=old-value", "{VARIABLE}=new-value")
   

5. Re-display configuration with updated value

6. Return to confirmation prompt (user can edit more or accept)

For PLF_PODMAN_MACHINE edits (macOS only):

If user wants to use a different Podman machine, list available machines:

podman machine ls --format "{{.Name}}"

Display options:

Podman Machine Selection:
  0. Create new 'k3d' machine (recommended for isolation)
  1. k3d (if exists)
  2. podman-machine-default
  3. [other existing machines...]

Enter machine name or number:

If user selects "0. Create new":

podman machine init k3d --cpus 4 --memory 16384 --now
podman system connection default k3d

Then run ./bin/plf doctor --fix to configure SSH access.

After confirmation, proceed to prerequisites check.

Step 0b: Prerequisites Check

Check manifest for cached tool verification:

grep "^tools_verified:" .snow-utils/snow-utils-manifest.md 2>/dev/null

If tools_verified: exists with a date: Skip tool checks, continue to Step 0c.

Otherwise, run prerequisite check:

./bin/plf doctor

The doctor command checks:

• Required tools: podman/docker, k3d, kubectl, uv
• Podman machine status (macOS only)
• SSH config for Podman VM access (macOS only)
• Port availability: 19000 (RustFS), 19001 (RustFS Console), 18181 (Apache Polaris), 6443 (k3d API)

With --fix flag (macOS with Podman):

• Creates Podman machine (k3d) if it doesn't exist
• Starts Podman machine if stopped
• Kills gvproxy if blocking port 19000 (required by RustFS)
• Sets up SSH config for Podman VM access

Note: doctor --fix is now fully automatic - it creates and configures the Podman machine without manual intervention. See docs/podman-setup.md for advanced configuration options.

If any tool is missing, stop and provide installation instructions from the Prerequisites table above.

STOP: Do not proceed until all prerequisites pass.

After all tools verified, update manifest:

grep -q "^tools_verified:" .snow-utils/snow-utils-manifest.md 2>/dev/null || \
  echo "tools_verified: $(date +%Y-%m-%d)" >> .snow-utils/snow-utils-manifest.md 2>/dev/null || true

Step 0c: Detect or Initialize Manifest

CRITICAL: Work Directory Check

BEFORE any manifest operations, verify we are NOT in the skill directory:

if [ -f "SKILL.md" ] && [ -d "src/polaris_local_forge" ]; then
  echo "ERROR: Currently in skill directory (read-only source)"
  echo "       Cannot create .snow-utils/ here"
  exit 1
fi

If in skill directory, STOP and ask:

⚠️  You're in the skill source directory (read-only).

Where would you like to create your Apache Polaris workspace?

  1. Create new directory: ~/polaris-dev
  2. Specify custom path: ___________

Enter choice [1]:

STOP: Wait for user input. NEVER proceed with manifest operations in skill directory.

After user provides work directory:

1. Create the directory if needed: mkdir -p <path>
2. Change to that directory: Use Shell working_directory parameter
3. THEN proceed with manifest detection below

---

Remote Manifest URL Detection

If the user provides a URL (in their prompt or pasted), detect and normalize it before local manifest detection:

Supported URL patterns and translation rules:

• GitHub blob: https://github.com/{owner}/{repo}/blob/{branch}/{path} -> replace host with raw.githubusercontent.com and remove /blob/ segment
• GitHub raw: https://raw.githubusercontent.com/... -> use as-is
• GitHub gist: https://gist.github.com/{user}/{id} -> append /raw if not already present
• Any other HTTPS URL ending in .md -> use as-is

After translating, show user and confirm:

Found manifest URL. Download URL:
  <translated_raw_url>

Download to current directory as <filename>? [yes/no]

STOP: Wait for user confirmation.

If yes:

curl -fSL -o <filename> "<translated_raw_url>"

After successful download, continue with local manifest detection below.

---

Check for existing manifest:

WORKING_MANIFEST=""
SHARED_MANIFEST=""

if [ -f .snow-utils/snow-utils-manifest.md ]; then
  WORKING_MANIFEST="EXISTS"
  WORKING_STATUS=$(grep "^\*\*Status:\*\*" .snow-utils/snow-utils-manifest.md | head -1)
  echo "Working manifest: .snow-utils/snow-utils-manifest.md (${WORKING_STATUS})"
fi

for f in *-manifest.md; do
  [ -f "$f" ] && grep -q "## shared_info\|CORTEX_CODE_INSTRUCTION" "$f" 2>/dev/null && {
    SHARED_MANIFEST="$f"
    echo "Shared manifest: $f"
  }
done

Decision matrix:

Working Manifest	Shared Manifest	Action
None	None	Fresh start -> Step 0e
None	Exists	Copy shared to .snow-utils/ -> Step 0d
Exists (REMOVED)	None	Replay Flow (reuse existing config)
Exists (COMPLETE)	None	Ask user: re-run, reset, or skip
Exists (IN_PROGRESS)	None	Resume Flow -- run pending steps from CLI Reference
Exists	Exists	Conflict -- ask user which to use

If BOTH manifests exist, show:

Found two manifests:

  1. Working manifest: .snow-utils/snow-utils-manifest.md
     Status: <WORKING_STATUS>

  2. Shared manifest: <SHARED_MANIFEST>
     (contains resource definitions from another setup)

Which manifest should we use?
  A. Resume working manifest (continue where you left off)
  B. Start fresh from shared manifest (backup working, adapt values)
  C. Cancel

STOP: Wait for user choice.

If using example manifest (user says "get started with apache polaris using example manifest"):

NOTE: This assumes the Work Directory Check above has passed. The example manifest is copied FROM the skill's example-manifests/ directory TO the user's work directory.

# In user's work directory (NOT skill directory!)
mkdir -p .snow-utils && chmod 700 .snow-utils
cp <SKILL_DIR>/example-manifests/polaris-local-forge-manifest.md .snow-utils/snow-utils-manifest.md
chmod 600 .snow-utils/snow-utils-manifest.md

Then proceed to Step 0d to check for adaptive markers.

Step 0d: Shared Manifest Adapt-Check

ALWAYS run this step when using a shared or example manifest. Prompt user ONLY if # ADAPT: markers are found.

ADAPT_COUNT=$(grep -c "# ADAPT:" .snow-utils/snow-utils-manifest.md 2>/dev/null)
echo "ADAPT markers found: ${ADAPT_COUNT}"

If ADAPT_COUNT > 0 (markers found):

Extract all values with # ADAPT: markers and present to user:

Manifest Value Review
─────────────────────
The following values can be customized for your environment:

  Setting                       Default Value              Marker
  ────────────────────────────  ─────────────────────────  ──────────────────────
  PLF_CONTAINER_RUNTIME:        podman                     # ADAPT: podman or docker
  K3D_CLUSTER_NAME:             (project directory name)   # ADAPT: customizable
  POLARIS_REALM:                POLARIS                    # ADAPT: customizable
  PLF_POLARIS_S3_BUCKET:        polaris                    # ADAPT: customizable
  PLF_POLARIS_CATALOG_NAME:     polardb                    # ADAPT: customizable
  PLF_POLARIS_PRINCIPAL_NAME:   super_user                 # ADAPT: customizable
  KUBECONFIG:                   (derived from cluster name)
  KUBECTL_PATH:                 (derived from cluster name)

Options:
  1. Accept all defaults (recommended for first-time setup)
  2. Edit a specific value
  3. Cancel

STOP: Wait for user choice.

Choice	Action
1 -- Accept all	Proceed with defaults
2 -- Edit specific	Ask which value, update manifest and .env in-place, re-display
3 -- Cancel	Stop

If user changes PLF_CONTAINER_RUNTIME: Update .env with the new value. If switching to Docker, clear PLF_PODMAN_MACHINE from .env.

If user changes K3D_CLUSTER_NAME: Automatically update derived values (KUBECONFIG, KUBECTL_PATH, resource table row 1) in the manifest. Also update .env with the new cluster name.

If user changes POLARIS_REALM or any PLF_POLARIS_* value: Update both the manifest and .env.

If ADAPT_COUNT = 0 (no markers): Proceed silently with values as-is.

Step 0e: Initialize Manifest

Use the --with-manifest flag to initialize the manifest along with the workspace:

./bin/plf init --with-manifest

This creates .snow-utils/snow-utils-manifest.md with proper permissions (chmod 700 for directory, chmod 600 for file).

If already initialized, you can add the manifest separately:

./bin/plf init --with-manifest --force

Step 1: Environment Setup

SHOW -- what we're about to do:

Set up the lightweight Python environment for querying and exploration. This creates a virtual environment and installs query dependencies (duckdb, pyiceberg==0.10.0, boto3, pandas) from the workspace pyproject.toml. Note: PyIceberg is pinned to 0.10.0 for Polaris compatibility (0.11.0 has REST API issues). The infrastructure CLI runs from the skill directory separately.

STOP: Wait for user confirmation before proceeding.

DO:

uv python pin 3.12
uv venv
uv sync --all-extras

SUMMARIZE:

Environment ready. Python venv created with query/notebook dependencies. Configuration loaded from .env.

Step 2: Full Setup

SHOW -- what we're about to do (tell user this):

I'll set up the complete Apache Polaris environment. This takes 2-5 minutes.

6 steps:

1. Pre-check & start Podman (~5s)
2. Generate configs (~10s)
3. Create k3d cluster (~30s)
4. Wait for RustFS & PostgreSQL (~60-90s)
5. Deploy Apache Polaris (~60s)
6. Setup catalog & verify (~35s)

Services will be available at:

• Apache Polaris API: localhost:18181
• RustFS S3: localhost:19000
• RustFS Console: localhost:19001

Ready to proceed?

STOP: Ask user for execution preference:

How would you like to proceed?

Present using AskQuestion tool with radio options:

Option	Label	Description
run_all	Run all (recommended)	Execute all setup steps automatically
step_by_step	Step by step	Pause after each step for confirmation

CRITICAL: Track EXECUTION_MODE for the rest of the session:

EXECUTION_MODE = user's choice ("run_all" or "step_by_step")

Based on user choice:

• If "run_all": Execute ALL steps below without pausing. NO confirmation prompts. Only stop on actual errors.
• If "step_by_step": Run each step, show output, then ask "Continue?" before the next step.

IMPORTANT: Once user selects "run_all", ALL subsequent stopping points in this session are SKIPPED. The skill proceeds automatically through Steps 1-8 without asking for confirmation. This includes:

• Prerequisite checks (auto-proceed if passing)
• Each CLI command execution (run immediately)
• Manifest updates (update immediately)

Only stop for:

1. Fatal errors that prevent continuation
2. Explicit destructive operations (teardown, purge) that were NOT part of the original setup flow

NOTE: Cortex Code will still show its command approval dialog (platform security feature). Tell user: "Select option 5 ('Allow using plf for this session') on the first plf command to avoid approval prompts for remaining steps."

DO -- Run each step as a SEPARATE command (REQUIRED for Cortex Code visibility):

./bin/plf doctor --fix                        # Step 1: Pre-check, auto-starts Podman
./bin/plf init --with-manifest                # Step 2: Copy templates, create dirs, init manifest

The init --with-manifest command creates the .snow-utils/snow-utils-manifest.md file with the resources tracking table.

Continue with remaining setup steps:

./bin/plf prepare                             # Step 3: Generate k8s configs
./bin/plf cluster create                      # Step 4: Create k3d cluster
./bin/plf cluster wait --tags bootstrap       # Wait for RustFS + PostgreSQL
./bin/plf polaris deploy                      # Step 5: Deploy Apache Polaris to cluster
./bin/plf cluster wait --tags polaris         # Wait for Apache Polaris + bootstrap job
./bin/plf polaris bootstrap                   # Step 6: Bootstrap Apache Polaris (create principal)
./bin/plf catalog setup                       # Step 7: Setup demo catalog
./bin/plf catalog verify-sql                  # Step 8: Verify with DuckDB

CRITICAL RULES:

• HONOR EXECUTION_MODE: If user selected "run_all", DO NOT ask for confirmation between steps. Proceed automatically.
• Run all mode: Execute full sequence, NO prompts between steps. Just run each command and report results.
• Step by step mode: Pause after each step, ask "Continue? [yes/no]"
• Run EACH command above as a SEPARATE bash invocation (for Cortex Code visibility)
• NEVER combine multiple commands (hides output from Cortex Code)
• NEVER use podman machine start directly (use doctor --fix)
• DESTRUCTIVE COMMANDS (teardown, delete, cleanup, purge): ALWAYS STOP and ask user for explicit confirmation BEFORE running. After user confirms, pass --yes to skip CLI's interactive prompt (CLI prompts don't work in non-interactive shell)
• FLAG USAGE: See Flag Usage Patterns for comprehensive guidance on when to use --force and --yes
• TEARDOWN AFTER SETUP: Even if user completes setup in this session and then asks to teardown, you MUST still present the teardown confirmation dialog with options. Session context does NOT bypass destruction confirmation. See Teardown Flow.
• DO NOT re-ask work directory, runtime, cluster name, etc. after user has already answered them

After setup completes, set the scoped cluster environment for this session:

export KUBECONFIG="$(pwd)/.kube/config"
export PATH="$(pwd)/bin:$PATH"
set -a && source .env && set +a

Update manifest after each successful step (RESILIENCE PATTERN):

After each step completes successfully, update the manifest resource row status from PENDING to DONE:

# Example: After cluster create succeeds, update row 1
# Use Edit/StrReplace tool to change: | 1 | k3d cluster | infrastructure | PENDING |
# To: | 1 | k3d cluster | infrastructure | DONE |

If interrupted mid-flow, the manifest preserves progress:

• Overall Status stays IN_PROGRESS
• Completed resources show DONE
• Pending resources show PENDING
• Resume Flow picks up from first PENDING resource

After ALL steps complete, update manifest status to COMPLETE:

# Use Edit/StrReplace tool to change: **Status:** IN_PROGRESS
# To: **Status:** COMPLETE

Verify manifest was written:

cat .snow-utils/snow-utils-manifest.md

Expected: all 7 resource rows show DONE, Status shows COMPLETE.

SUMMARIZE:

Setup complete. Cluster ${K3D_CLUSTER_NAME} running with Apache Polaris, RustFS, and PostgreSQL. Catalog ${PLF_POLARIS_CATALOG_NAME} created with principal ${PLF_POLARIS_PRINCIPAL_NAME}. Credentials saved to work/principal.txt. Manifest: .snow-utils/snow-utils-manifest.md

Step 3: Verification

SHOW -- what we're about to do:

Verify the entire setup by running a DuckDB SQL script that:

1. Connects to Apache Polaris REST catalog at http://localhost:18181
2. Authenticates using the principal credentials from work/principal.txt
3. Creates a test Iceberg table on RustFS
4. Queries the table to confirm read/write works

This confirms: Apache Polaris API, RustFS S3, catalog, principal, and RBAC grants are all working.

DO:

./bin/plf catalog verify-sql

Check the result:

• If SUCCEEDED: Continue to Step 3a (generate notebook)
• If FAILED: Check Troubleshooting section. Common issues: Apache Polaris not ready, RustFS not accessible, principal credentials invalid.

SUMMARIZE (on success):

Verification passed! DuckDB successfully queried Iceberg tables via Apache Polaris REST catalog on RustFS.

Step 3a: Generate Notebook

SHOW -- what we're about to do:

Generate a Jupyter notebook for interactive Apache Polaris and Iceberg exploration. The notebook is pre-configured with your catalog connection details and principal credentials, ready to run.

DO:

./bin/plf catalog generate-notebook

SUMMARIZE:

Notebook generated at notebooks/verify_polaris.ipynb. Open it with jupyter notebook notebooks/verify_polaris.ipynb for interactive exploration.

Step 4: Summary

SUMMARIZE -- Setup Complete:

Output the following summary (substitute actual values from .env). Remember: blank line before AND after the code block.

Apache Polaris Local Forge -- Setup Complete!

Service URLs:
  Apache Polaris API:     http://localhost:18181
  RustFS S3:       http://localhost:19000
  RustFS Console:  http://localhost:19001

Catalog:
  Name:      ${PLF_POLARIS_CATALOG_NAME}
  S3 Bucket: ${PLF_POLARIS_S3_BUCKET}
  Principal: ${PLF_POLARIS_PRINCIPAL_NAME}

Credentials:
  RustFS:    admin / password
  Apache Polaris:   See work/principal.txt
  Bootstrap: See k8s/polaris/.bootstrap-credentials.env

Next steps:
  jupyter notebook notebooks/verify_polaris.ipynb  # Interactive exploration
  ./bin/plf catalog verify-sql                        # Re-run verification
  ./bin/plf catalog explore-sql                       # Interactive DuckDB SQL

Shell setup:
  export KUBECONFIG=$(pwd)/.kube/config
  export PATH=$(pwd)/bin:$PATH
  set -a && source .env && set +a   # AWS config for RustFS
  kubectl get pods -n polaris
  aws s3 ls

Manifest: .snow-utils/snow-utils-manifest.md

Update manifest status to COMPLETE.

Catalog-Only Flows

These flows operate on the catalog without rebuilding the cluster.

Catalog Setup (cluster must be running)

Trigger: "setup catalog only", "create catalog"

./bin/plf catalog setup

Catalog Cleanup

Trigger: "cleanup catalog", "remove catalog"

STOP: Confirm with user before executing.

./bin/plf catalog cleanup --yes

Updates manifest status to REMOVED. Generated files in work/ are preserved.

Catalog Reset

Trigger: "reset polaris catalog", "recreate catalog"

STOP: Confirm with user before executing.

./bin/plf catalog cleanup --yes
./bin/plf catalog setup

Runs cleanup + setup. Generates new principal.txt (new credentials).

Full Catalog Reset

Trigger: "full catalog reset", "purge polaris database"

STOP: This is destructive. Confirm with user.

./bin/plf polaris reset --yes
./bin/plf catalog setup

Purges the entire Apache Polaris database and recreates from scratch.

Teardown Flow

Trigger: "teardown polaris", "delete everything", "clean up", "cleanup", "remove cluster"

CRITICAL - ALWAYS CONFIRM, NO EXCEPTIONS:

Even if user just completed setup:all in this same session, you MUST STOP and confirm before any destruction.

STOP - PRESENT OPTIONS TO USER:

Before running ANY teardown command, present this dialog:

Teardown Options:

  1. Cluster only (default)
     - Deletes k3d cluster, Polaris, RustFS, PostgreSQL
     - Keeps all local files for quick replay
     - Use `setup:replay` to restore

  2. Cluster + generated files
     - Deletes cluster AND cleans generated files
     - Use `setup:all` for fresh start

  CLEANUP_PATHS: .kube work k8s scripts bin notebooks datasets .env .aws .envrc .gitignore .venv pyproject.toml uv.lock

  NOTE: .snow-utils/ is ALWAYS preserved. Replay is possible after BOTH options
        unless you manually delete .snow-utils/snow-utils-manifest.md

Which option? [1/2] (or 'cancel' to abort)

Wait for user response - do NOT proceed without explicit selection.

After user selects option:

Option 1 (cluster only):

./bin/plf teardown --yes

Option 2 (cluster + files):

# CLEANUP_PATHS from Taskfile.yml - keep in sync!
./bin/plf teardown --yes
rm -rf .kube work k8s scripts bin notebooks datasets .env .aws .envrc .gitignore .venv pyproject.toml uv.lock

The --yes flag is passed because CLI prompts don't work in non-interactive shell, NOT to skip user confirmation from the agent.

After completion, output EXACTLY this message (DO NOT list individual commands or resources):

Teardown complete.

.snow-utils/ preserved for audit and replay. 

Run **replay** anytime to restore.

DO NOT list individual CLI commands. The agent can run replay when user asks.

Updates manifest status to REMOVED.

Podman Machine (macOS)

On macOS with Podman, teardown stops the Podman machine by default when using --yes (fully releases ports 19000, 19001, 18181). To keep Podman running:

./bin/plf teardown --yes --no-stop-podman

Replay Flow

Trigger: "replay polaris local forge", "recreate environment"

When manifest has Status: REMOVED:

1. Read config values from manifest
2. Show replay plan to user
3. On confirmation, execute Steps 1-10
4. Reuse existing .env and work directory layout
5. Re-activate scoped environment (KUBECONFIG, PATH for kubectl)
6. Regenerate catalog-level files (new principal credentials)
7. Update manifest to COMPLETE

L2C (Local to Cloud) Migration

WORKFLOW OVERVIEW: Migrate local Polaris Iceberg tables to AWS S3 and register as Snowflake External Iceberg Tables.

FORBIDDEN ACTIONS -- NEVER DO THESE:

• NEVER run aws CLI commands for cloud operations without scrubbing the local RustFS environment first -- the .env sets AWS_CONFIG_FILE, AWS_SHARED_CREDENTIALS_FILE, AWS_ENDPOINT_URL, and AWS_ACCESS_KEY_ID to local RustFS (localhost:19000). Before running ANY aws sts, aws s3, aws sso, or other cloud-targeted AWS commands, unset these variables so the AWS CLI uses real credentials from ~/.aws/. See Scrubbed AWS Environment for the exact pattern
• NEVER run snow sql, snowflake.connector.connect(), or any direct Snowflake SQL/Python -- the .env SNOWFLAKE_DEFAULT_CONNECTION_NAME differs from the Snowflake CLI default connection. ALL Snowflake operations MUST go through ./bin/plf l2c commands which use the correct connection name, role, database, and schema from .env + l2c-state.json
• NEVER guess or manually construct Snowflake connection parameters (connection_name, role, database, schema) -- these are resolved and stored in .env, .snow-utils/l2c-state.json, and .snow-utils/snow-utils-manifest.md. Read from these files if needed, but prefer ./bin/plf l2c commands which resolve them automatically
• NEVER run L2C commands without local Polaris cluster running
• NEVER proceed without AWS credentials configured (profile or SSO)
• NEVER skip dry-run verification before actual migration
• NEVER hardcode AWS/Snowflake credentials in commands
• NEVER use sed/awk for .env edits -- use Edit/StrReplace tool
• NEVER skip user confirmation for destructive operations
• NEVER guess CLI options -- run --help first
• NEVER run L2C commands if l2c-state.json shows incomplete setup
• NEVER proceed with sync/register without successful setup completion

INTERACTIVE PRINCIPLE: This L2C workflow is designed to be interactive. At every decision point, ASK the user and WAIT for their response before proceeding.

DUAL-MODE EXECUTION:

• CLI Mode: Direct plf l2c commands for setup/sync/register operations
• Notebook Mode: Interactive verification via l2c_workbook.ipynb with skill actions

MANIFEST FILE: .snow-utils/snow-utils-manifest.md (exact path, always .md)

L2C SOURCE OF TRUTH -- NEVER GUESS, ALWAYS READ:

All L2C configuration is resolved and stored in these files. Cortex MUST read from them instead of guessing, running exploratory SQL, or constructing connection parameters manually.

File	Contains	Used By
.env	SNOWFLAKE_DEFAULT_CONNECTION_NAME, SNOWFLAKE_USER, SNOWFLAKE_ACCOUNT, L2C_AWS_PROFILE	All plf l2c commands for connection resolution
.snow-utils/l2c-state.json	sa_role, database, schema, per-table sync/register state, S3 bucket, AWS region	All plf l2c commands for operations
.snow-utils/snow-utils-manifest.md	L2C Snowflake section (<!-- START -- l2c:snowflake -->): Admin Role, SA_ROLE, Catalog Integration, External Volume, Database, Schema, per-resource statuses (DONE/PENDING/REMOVED)	plf l2c setup snowflake reads admin_role; audit trail and replay
~/.snowflake/connections.toml	Named connection auth details (account URL, auth method)	Referenced by SNOWFLAKE_DEFAULT_CONNECTION_NAME

L2C COMMAND ROUTING:

When user requests L2C operations, route DIRECTLY to the corresponding ./bin/plf l2c command. Do NOT improvise with snow sql, snowflake.connector, or manual SQL.

User Intent	CLI Command
"check migration status", "show l2c status"	./bin/plf l2c status
"list tables for migration", "what tables available"	./bin/plf l2c inventory
"migrate tables", "run full migration"	./bin/plf l2c migrate --yes
"sync to S3", "copy to cloud"	./bin/plf l2c sync --yes
"register in Snowflake", "create external tables"	./bin/plf l2c register --yes
"refresh Snowflake tables", "update metadata"	./bin/plf l2c refresh --yes
"clear tables", "reset migration state"	./bin/plf l2c clear --yes
"full cleanup", "teardown L2C"	./bin/plf l2c cleanup --yes
"verify Snowflake tables", "check row counts"	./bin/plf l2c status
"preview migration", "dry run"	./bin/plf l2c sync --dry-run
"setup L2C", "provision AWS and Snowflake"	./bin/plf l2c setup
"mutated data locally, re-sync"	./bin/plf l2c sync --yes (snapshot-aware auto-escalates to force)
"demo reset, re-migrate"	./bin/plf l2c clear --yes then ./bin/plf l2c migrate --yes
"force re-upload everything"	./bin/plf l2c sync --force --yes

Prerequisites for L2C

STOP: Check prerequisites before proceeding.

Required Tools:

Tool	Purpose	Verification
AWS CLI v2	AWS S3 access	aws --version
Snowflake CLI	Snowflake connection	snow --version
Local Polaris	Source catalog	./bin/plf cluster status

Required Configuration:

1. AWS Credentials: Either AWS profile or SSO configured FOR REAL AWS S3
2. Snowflake Connection: Named connection in ~/.snowflake/connections.toml
3. Local Polaris: Running cluster with data to migrate

⚠️ CRITICAL: AWS Credential Resolution Logic

Local Polaris uses RustFS (localhost:19000) with static credentials (admin/password) L2C Migration uses this exact credential resolution order:

1. CLI flag: --aws-profile <profile-name>
2. Environment variable: L2C_AWS_PROFILE=<profile-name>
3. Default credential chain: Uses boto3.Session() default behavior
   • Environment variables (AWS_ACCESS_KEY_ID, etc.)
   • AWS SSO credentials
   • AWS CLI profile 'default'
   • EC2 instance profile (if running on EC2)

Profile Resolution Function: _resolve_profile(aws_profile) -> aws_profile or os.environ.get("L2C_AWS_PROFILE") or "default"

Verify AWS credentials (scrub local RustFS env first):

⚠️ CRITICAL: The .env sets AWS_CONFIG_FILE, AWS_SHARED_CREDENTIALS_FILE, AWS_ENDPOINT_URL, and AWS_ACCESS_KEY_ID to local RustFS (localhost:19000). You MUST scrub these variables before running any aws command targeting real AWS.

Scrubbed AWS Environment Pattern:

# Scrub local RustFS env vars, then run real AWS commands
env -u AWS_CONFIG_FILE -u AWS_SHARED_CREDENTIALS_FILE -u AWS_ENDPOINT_URL \
    -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN \
    -u AWS_DEFAULT_REGION -u AWS_REGION -u AWS_PROFILE -u AWS_DEFAULT_PROFILE \
    aws sts get-caller-identity

Use this same env -u ... prefix for ALL cloud AWS commands:

# Verify real AWS identity
env -u AWS_CONFIG_FILE -u AWS_SHARED_CREDENTIALS_FILE -u AWS_ENDPOINT_URL \
    -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN \
    -u AWS_DEFAULT_REGION -u AWS_REGION -u AWS_PROFILE -u AWS_DEFAULT_PROFILE \
    aws sts get-caller-identity

# SSO login
env -u AWS_CONFIG_FILE -u AWS_SHARED_CREDENTIALS_FILE -u AWS_ENDPOINT_URL \
    -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN \
    -u AWS_DEFAULT_REGION -u AWS_REGION -u AWS_PROFILE -u AWS_DEFAULT_PROFILE \
    aws sso login --profile <your-profile>

# List real S3 buckets
env -u AWS_CONFIG_FILE -u AWS_SHARED_CREDENTIALS_FILE -u AWS_ENDPOINT_URL \
    -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN \
    -u AWS_DEFAULT_REGION -u AWS_REGION -u AWS_PROFILE -u AWS_DEFAULT_PROFILE \
    aws s3 ls

Note: ./bin/plf l2c commands handle this automatically via the scrubbed_aws_env() context manager in Python. The env -u pattern above is the shell equivalent for direct aws CLI usage.

Alternatively, use ./bin/plf l2c setup --dry-run which scrubs automatically and shows the resolved AWS account, region, and planned resources.

If using a specific AWS profile, set it in .env:

L2C_AWS_PROFILE=<your-profile>

Check Snowflake connection:

snow connection test

STOP: Do not proceed until all prerequisites pass.

Step 1: L2C Prerequisites Check

STOP: Wait for user confirmation of AWS/Snowflake setup.

Ask user:

I need to verify your L2C migration prerequisites:

1. Do you have AWS credentials configured (profile or SSO)?
2. Do you have a Snowflake connection configured?
3. Is your local Polaris cluster running with data?

AWS Credential Isolation: L2C uses scrubbed_aws_env() context manager to temporarily clear RustFS environment variables (AWS_ACCESS_KEY_ID=admin, AWS_ENDPOINT_URL=localhost:19000) so that boto3 can access your real AWS credentials without interference.

Please confirm all three are ready before proceeding.

After user confirms, verify local Polaris:

./bin/plf cluster status

If cluster not running:

STOP: Direct user to run the main workflow first to set up local Polaris.

Step 2: L2C Setup (AWS + Snowflake)

⚠️ IMPORTANT: Credential Context Switch

L2C setup will now use your REAL AWS credentials (profile/SSO), NOT the RustFS static credentials. Make sure your AWS profile is configured for the account where you want to create S3 buckets.

SHOW: Preview the L2C setup command:

./bin/plf l2c setup --dry-run

This will:

• Create REAL AWS S3 bucket for Iceberg data (using your AWS profile/SSO)
• Set up Snowflake database and schema
• Configure External Volume in Snowflake
• Generate L2C configuration in l2c-state.json

STOP: Wait for user approval before executing setup commands.

DO: Execute the setup:

./bin/plf l2c setup

SUMMARIZE: Explain what was created and show the l2c-state.json configuration (mask sensitive values).

Step 3: L2C Migration (Sync + Register)

SHOW: Preview the migration process:

./bin/plf l2c inventory

This shows all tables available for migration.

STOP: Wait for user to review tables and confirm migration.

SHOW: Preview the sync operation:

./bin/plf l2c sync --dry-run

This will:

• Copy Iceberg metadata and data files to S3
• Rewrite metadata paths for S3 locations
• Preserve table structure and partitioning

STOP: Wait for user approval before executing migration.

DO: Execute the migration:

./bin/plf l2c sync

SHOW: Preview the registration:

./bin/plf l2c register --dry-run

This will create External Iceberg Tables in Snowflake.

STOP: Wait for user approval before executing registration.

DO: Execute the registration:

./bin/plf l2c register

SUMMARIZE: Show migration status and registered tables:

./bin/plf l2c status

Step 4: L2C Verification (Notebook Actions)

STOP: Wait for user to choose verification method.

Offer user two verification options:

How would you like to verify the L2C migration?

1. CLI Verification - Quick command-line checks
2. Notebook Verification - Interactive Jupyter notebook with detailed analysis

Option 1: CLI Verification

SHOW: Preview verification commands:

# Check migration status (shows sync/register state per table)
./bin/plf l2c status

# Inventory local tables available for migration
./bin/plf l2c inventory

DO: Execute verification commands after user approval. Do NOT run snow sql directly -- plf l2c status reports Snowflake table state including row counts.

Option 2: Notebook Verification

SHOW: Preview notebook actions available:

The enhanced user-project/notebooks/l2c_workbook.ipynb provides comprehensive interactive verification:

Core Capabilities:

• Local Inventory: Discover tables available for migration with metadata
• Migration Status: Real-time sync and registration progress tracking
• Data Verification: Compare row counts between local Polaris and Snowflake
• Cloud Verification: S3 object counting and metadata validation
• Incremental Updates: Zero-downtime data mutation workflows
• Reset/Reload: Clean slate for iterative development

Built-in Utility Functions:

• AWS credential isolation (mirrors CLI scrubbed_aws_env())
• DuckDB + Polaris REST connection management
• Snowflake connection with proper role/database context
• S3 object counting for sync verification
• PyIceberg compatibility handling (version 0.10.0)

DO: Guide user to open and execute the notebook:

# Open notebook in Jupyter
jupyter notebook user-project/notebooks/l2c_workbook.ipynb

# Or open in Cursor for integrated experience
cursor user-project/notebooks/l2c_workbook.ipynb

SUMMARIZE: The notebook provides a complete L2C migration experience with interactive verification, utility functions, and step-by-step guidance through the entire local-to-cloud workflow.

L2C Maintenance Operations

Incremental Updates

SHOW: Preview incremental sync:

./bin/plf l2c sync --incremental

STOP: Wait for user approval.

DO: Execute incremental sync and register updates.

Reset and Reload

SHOW: Preview reset operation:

./bin/plf l2c cleanup

This will remove cloud resources but preserve local data.

STOP: This is destructive. Confirm with user.

DO: Execute cleanup and optionally reload demo data.

Status Monitoring

Check L2C status anytime:

./bin/plf l2c status

L2C Troubleshooting

Common Issues:

Issue	Cause	Solution
AWS credentials error	Profile/SSO not configured	Run aws configure sso or set profile
Snowflake connection failed	Connection not configured	Run snow connection add
S3 bucket creation failed	Permissions or naming conflict	Check AWS permissions, try different bucket name
Table registration failed	Snowflake permissions	Verify External Volume and database permissions
Boto3 uses RustFS credentials	Local .env contaminating AWS calls	Use scrubbed environment pattern (see below)
"NoCredentialsError" during L2C	AWS profile expired or not found	Re-authenticate with aws sso login

AWS Credential Isolation Issues:

The most common L2C issue is credential contamination between local RustFS and real AWS. The project .env file sets:

• AWS_ACCESS_KEY_ID=admin (for RustFS)
• AWS_ENDPOINT_URL=http://localhost:19000 (for RustFS)
• AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE (pointing to local files)

These variables interfere with real AWS operations. L2C commands handle this automatically, but manual aws CLI usage requires the scrubbed environment pattern.

Debug Commands:

# Check L2C configuration
cat .snow-utils/l2c-state.json

# Verify real AWS access (scrub RustFS env first -- see "Scrubbed AWS Environment Pattern" above)
env -u AWS_CONFIG_FILE -u AWS_SHARED_CREDENTIALS_FILE -u AWS_ENDPOINT_URL \
    -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN \
    -u AWS_DEFAULT_REGION -u AWS_REGION -u AWS_PROFILE -u AWS_DEFAULT_PROFILE \
    aws sts get-caller-identity

# Or use PLF which scrubs automatically
./bin/plf l2c setup --dry-run

# Verify real S3 bucket access (scrubbed)
env -u AWS_CONFIG_FILE -u AWS_SHARED_CREDENTIALS_FILE -u AWS_ENDPOINT_URL \
    -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN \
    -u AWS_DEFAULT_REGION -u AWS_REGION -u AWS_PROFILE -u AWS_DEFAULT_PROFILE \
    aws s3 ls s3://<bucket-name>/

# Test Snowflake connection
snow connection test

# Check Polaris catalog
./bin/plf catalog query --sql "SHOW ALL TABLES"

Notebook-Specific Troubleshooting:

If using the L2C workbook (user-project/notebooks/l2c_workbook.ipynb), be aware that:

1. s3fs Cache Issues: The notebook clears s3fs.S3FileSystem.clear_instance_cache() to prevent credential conflicts
2. Environment Isolation: The notebook uses inline environment isolation similar to the CLI's scrubbed_aws_env() pattern
3. PyIceberg Compatibility: Ensure PyIceberg is pinned to version 0.10.0 for Polaris compatibility

Common Notebook Errors:

• "PUT is not a valid HttpMethod" → Downgrade PyIceberg to 0.10.0
• NoCredentialsError in boto3 calls → Check AWS profile authentication
• EndpointConnectionError → Verify local Polaris cluster is running

Apache Polaris API Queries

EXPERIMENTAL FEATURE: REST API query support via natural language is experimental. Currently supports read-only operations. Write operations (create, delete, update) will be added in a future phase.

AUTO-ACTIVATION: If user's current directory has .snow-utils/snow-utils-manifest.md with polaris-local-forge:, or .env with PLF_POLARIS_CATALOG_NAME, this is an active polaris-local-forge project. Skip all disambiguation and use plf api query (see PROJECT CONTEXT DETECTION above).

When to use: User asks to list, show, describe, or query catalogs, namespaces, tables, views, principals, roles, or grants.

IMPORTANT — GO DIRECTLY TO ANSIBLE:

• There is NO polaris CLI — do NOT run which polaris or polaris --help
• There is NO plf query or plf polaris subcommand
• Do NOT search for CLI tools — go DIRECTLY to the Ansible playbook
• Do NOT try curl or jq — Ansible handles OAuth and credentials automatically
• Construct the endpoint path dynamically based on user intent

How: Use ./bin/plf api query <endpoint> — the CLI handles Ansible, OAuth, and output formatting.

SEMANTIC UNDERSTANDING: Recognize intent variations — these all mean the same thing:

• "list catalogs" = "show me catalogs" = "what catalogs exist" = "get all catalogs"
• "show namespaces" = "list namespaces" = "namespaces in catalog" = "what namespaces"
• "describe table X" = "show table X" = "get table X details" = "table X schema"

CRITICAL DISAMBIGUATION — "namespaces" means POLARIS NAMESPACES:

• When user says "namespaces", "list namespaces", "show namespaces" — this means Apache Polaris catalog namespaces, NOT Kubernetes namespaces
• Do NOT run kubectl get ns or check Kubernetes namespaces
• Do NOT run curl to any endpoint — Ansible handles this
• Do NOT check if Polaris is running or look for services — assume setup is complete
• Go DIRECTLY to Ansible with endpoint /api/catalog/v1/{catalog}/namespaces

Why Ansible (not curl/jq)

• No extra dependencies — users don't need jq installed
• Credentials hidden — OAuth token retrieval uses no_log: true, never exposed
• Consistent pattern — same approach as existing catalog_setup.yml
• JSON output — Ansible returns structured JSON for easy formatting
• Reads .env — Automatically uses user's configured catalog name

Endpoint Construction (Generic Pattern)

The playbook accepts an endpoint parameter. Construct the endpoint path based on user intent:

Management API Endpoints (/api/management/v1/...):

User Intent	Endpoint Path
"list catalogs", "show catalogs"	/api/management/v1/catalogs
"show catalog {name}", "describe catalog {name}"	/api/management/v1/catalogs/{name}
"list principals", "show principals"	/api/management/v1/principals
"show principal {name}", "describe principal {name}"	/api/management/v1/principals/{name}
"list principal roles", "show my roles"	/api/management/v1/principal-roles
"show principal role {name}"	/api/management/v1/principal-roles/{name}
"list roles for principal {name}"	/api/management/v1/principals/{name}/principal-roles
"list catalog roles for {catalog}"	/api/management/v1/catalogs/{catalog}/catalog-roles
"show catalog role {role} in {catalog}"	/api/management/v1/catalogs/{catalog}/catalog-roles/{role}
"list grants for role {role} in {catalog}"	/api/management/v1/catalogs/{catalog}/catalog-roles/{role}/grants

Catalog API Endpoints (/api/catalog/v1/...):

User Intent	Endpoint Path
"list namespaces", "show namespaces"	/api/catalog/v1/{catalog}/namespaces
"show namespace {ns}", "describe namespace {ns}"	/api/catalog/v1/{catalog}/namespaces/{ns}
"list tables in {ns}", "show tables"	/api/catalog/v1/{catalog}/namespaces/{ns}/tables
"show table {table} in {ns}", "describe table {table}"	/api/catalog/v1/{catalog}/namespaces/{ns}/tables/{table}
"list views in {ns}", "show views"	/api/catalog/v1/{catalog}/namespaces/{ns}/views
"show view {view} in {ns}", "describe view {view}"	/api/catalog/v1/{catalog}/namespaces/{ns}/views/{view}

Endpoint Construction Rules

1. Parse user intent to identify the operation type (list/show/describe)
2. Extract entity names from the request (catalog, namespace, table, view, principal, role)
3. Use defaults when not specified:
   • {catalog} → use value from .env PLF_POLARIS_CATALOG_NAME (typically polardb)
   • {ns} → use default if not specified
4. Substitute placeholders in the endpoint path
5. Execute with the constructed endpoint

Execution Pattern

Use ./bin/plf api query <endpoint> — the CLI handles Ansible, OAuth, environment, and output formatting:

# List all catalogs
./bin/plf api query /api/management/v1/catalogs

# List namespaces in polardb catalog
./bin/plf api query /api/catalog/v1/polardb/namespaces

# Show table details
./bin/plf api query /api/catalog/v1/polardb/namespaces/default/tables/penguins

# List grants for a catalog role
./bin/plf api query /api/management/v1/catalogs/polardb/catalog-roles/admin/grants

# Text output instead of JSON
./bin/plf api query /api/management/v1/catalogs -o text

# Verbose mode for debugging
./bin/plf api query /api/management/v1/catalogs -v

Use Shell tool with working_directory set to <WORK_DIR>:

Shell(command="./bin/plf api query /api/catalog/v1/polardb/namespaces", working_directory="<WORK_DIR>")

Messaging Template

When running API queries, show clear progress:

**Querying Apache Polaris:** {intent description}

Running: `plf api query {endpoint}`

Example:

**Querying Apache Polaris:** list namespaces in polardb catalog

Running: `plf api query /api/catalog/v1/polardb/namespaces`

Output Formatting

Ansible returns JSON in result.json. Format as markdown table for the user:

**Namespaces in catalog `polardb`:**

| Namespace |
|-----------|
| default |
| analytics |

Export for Sharing Flow

Trigger phrases: "export manifest for sharing", "share polaris manifest"

Purpose: Create a portable copy of the manifest that another developer can use to replay the entire setup on their machine.

Precondition

Cortex Code MUST verify manifest has Status: COMPLETE or Status: REMOVED:

grep "^\*\*Status:\*\*" .snow-utils/snow-utils-manifest.md

If status is IN_PROGRESS, refuse with: "Cannot export incomplete setup. Complete the setup first or teardown to set status to REMOVED."

Export Steps

1. Read active manifest from .snow-utils/snow-utils-manifest.md

2. Read project_name from ## project_recipe section

3. Determine shared_by from machine username (fall back to asking user):

   SHARED_BY=$(whoami)
   echo "Exporting as: ${SHARED_BY}"
   

4. Ask user for export location:

   Export manifest for sharing:
   
     Filename: {project_name}-manifest.md
     Default location: ./ (project root)
   
   Save to [./]:
   

   STOP: Wait for user input.

5. Ask about container runtime:

   Container Runtime Export Option
   ───────────────────────────────
   Your setup used: {container_runtime from manifest}
   
   How should the exported manifest handle container runtime?
   
     1. Keep current value ({container_runtime}) - recipient reproduces exact environment
     2. Clear for auto-detection - recipient's CLI detects their runtime
   
   Choice [2]:
   

   STOP: Wait for user input. Default is 2 (auto-detect) for maximum flexibility.

6. If file already exists at target: Ask overwrite / rename with timestamp / cancel

7. Create export file with these transformations:

   • Inject <!-- CORTEX_CODE_INSTRUCTION --> HTML comment at top (see example-manifests/polaris-local-forge-manifest.md for format)
   • Add ## shared_info section with shared_by: {SHARED_BY}, shared_date, original_project_dir, notes
   • Preserve ## installed_skills with skill URL (enables Cortex self-installation)
   • Change **Status:** COMPLETE to **Status:** REMOVED
   • Add # ADAPT: customizable markers on user-customizable values
   • If user chose option 2: Set container_runtime: # Auto-detected by CLI
   • If user chose option 1: Keep container_runtime: {value}
   • Remove podman_machine line (always auto-detected by CLI)

Note: The exported file is in the project root, NOT in .snow-utils/. Skills only read from .snow-utils/snow-utils-manifest.md so the export is invisible to all skill flows.

Consuming Projects: Minimal Setup

A separate project that wants to query the Apache Polaris catalog needs only:

In the project directory:

• .env with:

POLARIS_URL=http://localhost:18181
AWS_ENDPOINT_URL=http://localhost:19000
AWS_ACCESS_KEY_ID=admin
AWS_SECRET_ACCESS_KEY=password
AWS_REGION=us-east-1
# From work/principal.txt in the polaris workspace:
POLARIS_REALM=POLARIS
CLIENT_ID=<from principal.txt>
CLIENT_SECRET=<from principal.txt>

• A notebook (.ipynb) or SQL scripts for querying
• pyproject.toml with query deps (copy user-project/pyproject.toml from the skill repo)

NOT needed: k8s manifests, ansible, polaris-local-forge CLI source.

Stopping Points

1. Step 0: Ask for workspace directory (if not detected)
2. Step 0a: Configuration review -- wait for user confirmation
3. Step 0b: If prerequisites missing (show install instructions)
4. Step 0c: Manifest detection (ask which to use if conflict)
5. Step 0d: Adapt-check (if shared manifest has # ADAPT: markers)
6. Step 1: Before environment setup (uv venv)
7. Step 2: Before full setup (show 6-step plan, wait for OK)
8. Step 3: Before verification (DuckDB test)
9. Step 3a: Before generating notebook
10. Catalog-only flows: Before cleanup/reset
11. Teardown: Before destructive operations

CLI Reference

All commands use the ./bin/plf wrapper script (created by init):

./bin/plf <command>

The wrapper sources .env automatically and handles all paths. Aliased as plf in tables below.

Command Execution

IMPORTANT: Run commands from the project directory using the Shell tool's working_directory parameter:

Shell(command="./bin/plf <COMMAND>", working_directory="<PROJECT_HOME>")

DO NOT use cd && ./bin/plf pattern - use working_directory instead for cleaner output.

The wrapper sources .env (created by init) which contains:

PROJECT_HOME=/path/to/your/project
K3D_CLUSTER_NAME=your-project-name
SKILL_DIR=/path/to/polaris-local-forge

Commands this skill executes:

Logical Command	Purpose
plf doctor	Check prerequisites
plf doctor --fix	Fix Podman/SSH issues
plf init	Initialize work directory
plf prepare	Generate configuration files
plf cluster create	Create k3d cluster
plf cluster wait	Wait for deployments
plf polaris deploy	Deploy Apache Polaris
plf polaris bootstrap	Create admin principal
plf catalog setup	Setup Iceberg catalog
plf catalog verify-sql	Verify with DuckDB
plf cluster delete --yes	Delete cluster (destructive)
plf teardown --yes	Full cleanup (destructive)
plf l2c setup	Provision AWS (S3, IAM) and Snowflake resources for L2C
plf l2c inventory	List local Iceberg tables available for migration
plf l2c status	Show migration status (sync/register state per table)
plf l2c sync --yes	Sync local Iceberg data to S3
plf l2c sync --dry-run	Preview sync without executing
plf l2c register --yes	Register External Iceberg Tables in Snowflake
plf l2c migrate --yes	Full migration (sync + register)
plf l2c refresh --yes	Refresh Snowflake external table metadata
plf l2c clear --yes	Clear table migration state (re-sync/re-register)
plf l2c cleanup --yes	Teardown L2C AWS and Snowflake resources

Recommended permission setting: When Cortex Code prompts for command approval:

Before init (uses uv):

• Option 5: "Allow using 'uv' for this session" - Recommended

After init (uses ./bin/plf wrapper):

• Option 5: "Allow using 'plf' for this session" - Recommended for "Run all" mode
• Option 4: "Always allow using 'plf'" - If you use this skill frequently

TIP: When running "Run all (recommended)" mode, select option 5 on the first plf command to avoid approval prompts for each of the 8+ setup steps.

This is safe because:

• You control the skill source code
• Commands are well-defined CLI operations
• uv only runs Python from your trusted project

Global options (before any subcommand):

Option	Description
--work-dir PATH	Working directory for generated files (default: skill directory)
--env-file PATH	Path to .env file (default: <work-dir>/.env)

OPTION NAMES (NEVER guess or invent options):

ONLY use options listed in the tables below. If a command fails with "No such option", run ./bin/plf <command> --help to see actual available options and use ONLY those. NEVER invent, abbreviate, or rename options.

--yes flag: Pass --yes ONLY after getting explicit user confirmation from the agent conversation. The flag skips the CLI's interactive prompt (which doesn't work in Cortex Code's non-interactive shell), but the agent MUST still ask the user for confirmation before running the command. All destructive commands support --dry-run to preview.

COMMAND NAMES (exact -- do NOT substitute):

• catalog setup -- NOT "catalog create", "catalog init"
• catalog cleanup -- NOT "catalog delete", "catalog remove"
• catalog verify-sql -- NOT "catalog verify", "verify-sql"
• catalog explore-sql -- NOT "catalog explore", "explore-sql"
• cluster create -- NOT "cluster setup", "cluster init"
• cluster delete -- NOT "cluster remove", "cluster destroy"

FORBIDDEN FILE EDIT COMMANDS (NEVER use for config files):

CRITICAL: NEVER use sed, awk, or bash string manipulation to edit .env, manifest files, or any configuration files. Use the Edit/StrReplace tool instead.

NEVER Use	Use Instead
sed -i 's/old/new/' .env	Edit/StrReplace tool
awk '{...}' file > newfile	Edit/StrReplace tool
echo "VAR=value" >> .env	Edit/StrReplace tool
cat > .env << EOF	Write tool (only for new files)

If ports are blocked (gvproxy error on port 19000):

Port 19000 is required by RustFS. On macOS with Podman, the gvproxy network process may hold this port. The doctor --fix command handles this automatically:

# Automatic fix (recommended)
./bin/plf doctor --fix

This will:

• Kill gvproxy processes blocking port 19000
• Stop and restart the Podman machine
• Set up SSH config for VM access

Manual fix (if automatic fails):

# Only use if doctor --fix doesn't work
pkill -f gvproxy
podman machine stop k3d
sleep 5
podman machine start k3d
./bin/plf doctor --fix  # Re-run to verify

After fixing ports, continue setup:

./bin/plf prepare
./bin/plf cluster create
./bin/plf cluster wait --tags bootstrap
./bin/plf polaris deploy
./bin/plf cluster wait --tags polaris
./bin/plf polaris bootstrap
./bin/plf catalog setup
./bin/plf catalog verify-sql

Doctor (Prerequisites Check)

Command	Description
plf doctor	Check prerequisites, tools, Podman machine, ports
plf doctor --fix	Auto-fix: create Podman machine if missing, start if stopped, kill gvproxy on port 19000, setup SSH
plf doctor --output json	Output status as JSON

Runtime

Command	Description
plf runtime detect	Detect container runtime (docker/podman/choice)
plf runtime detect --json	Output detection result as JSON (for agents)
plf runtime docker-host	Output DOCKER_HOST value for current runtime

Init & Prepare

Command	Description
plf init	Initialize project directory (.env, .envrc, .gitignore, directories)
plf init --force	Re-initialize, overwriting existing files
plf init --runtime docker|podman	Initialize with explicit runtime (skips interactive prompt)
plf prepare	Generate config files from templates (runs ansible prepare.yml)