| name | tableau-to-sigma |
| description | Convert a Tableau datasource or workbook into a Sigma data model and matching dashboard. Use when the user has a Tableau datasource, TDS file, or Tableau workbook and wants to recreate it in Sigma. Discovery, calc-field translation, data model + workbook creation via REST API, layout generation, and parity verification — driven by `scripts/*.rb`. |
| user-invocable | true |
Tableau → Sigma Conversion
⛔ STEP 0 — MANDATORY environment bootstrap (before any other step)
On the first run on a machine, run the ONE bootstrap command — idempotent,
non-interactive, no admin, never echoes credentials:
- macOS / Linux / Git Bash:
bash scripts/bootstrap.sh --workdir <WORK>
- Windows PowerShell:
powershell -ExecutionPolicy Bypass -File scripts\bootstrap.ps1 -WorkDir <WORK>
It verifies/activates/installs ruby + python3 (+ Pillow/numpy/requests) + node
pinned to 22 LTS (user-scoped — winget/scoop on Windows,
brew/rbenv/fnm-aware elsewhere; PATH persists to ~/.sigma-migration/path.sh
(bash) / USER PATH (Windows)), persists creds from env
vars (setup.rb --from-env), ends in a doctor run (doctor.json), and
writes the bootstrap sentinel (bootstrap.json).
intake.rb and migrate-tableau.rb REFUSE to start without sentinel +
doctor-green. NEVER hand-install a runtime or edit PATH yourself — if
bootstrap reports no admin-free route, surface its message to the user and
stop. Dry run: --check / -Check. Details: refs/environment.md.
⛔ STEP 1 — THE ONE PATH (there is a single entry point; do not improvise one)
After the bootstrap, always run the orchestrator — it chains every phase
(discover → gates → DM → workbook → layout → two-pass parity → cleanup) in one
process and self-gates:
ruby scripts/migrate-tableau.rb --workbook "<name>" --connection <id>
- Pass the Tableau
/#/views/… share URL straight to --workbook (quote it).
The orchestrator resolves the workbook LUID from the URL's contentUrl slug
in process. Display names diverge from slugs ("High Risk Bets" vs
HighRiskBets), so never hand-roll name:eq: REST filters or page the
workbook list yourself — that field-cost a 500-workbook scan. Give it the URL.
- Flag cheatsheet — use the EXACT flags (do not guess or grep
opts.on):
migrate-tableau.rb --out DIR (the workdir flag is --out, not --workdir);
validate-spec.rb <spec.json> --type dm|workbook (spec is a positional arg, no --spec);
put-layout.rb --workbook ID (not --workbook-id), reading SIGMA_API_TOKEN from the env.
- Do NOT hand-drive the per-phase scripts, and do NOT hand-author DM/workbook
JSON, UNLESS an orchestrator STOP message explicitly routes you there (e.g.
"CONVERTER STOP", the exit-4 workbook handoff). The script map is for
understanding and recovery, not a menu — à-la-carte scripts are the #1
way runs go inconsistent (the run-state audit silently no-ops when bypassed).
- "Done" is not something you decide — it is a file on disk. The migration
is complete only when
ruby scripts/verify-complete.rb --workdir <WORK>
exits 0 / prints ✅ DONE. A clean PASS 1 (exit 12) is NOT done — it means
"run --finalize". Never report success or write a completion summary before.
- This is a PRODUCTION migration, not a demo — regardless of who is watching
or why. The bar is EXACT parity against the same warehouse, always. At a
wall you have exactly two moves: follow the STOP/handoff the orchestrator
printed, or surface the blocker plainly and stop. NEVER a third path:
no "lighter"/"simplified"/"good-enough-for-the-demo" builds, no dropping
tiles/filters/calcs to look finished. Cutting fidelity to escape a struggle
is a worse failure than stopping. If a lighter scope is wanted, the
user decides that explicitly — you never volunteer it.
- An orchestrator STOP is an instruction to YOU (the agent), not a handoff to
a human — keep going. A routing exit (exit-4 workbook handoff, "CONVERTER
STOP") authorizes you to do the next step yourself: author the spec it names
and re-enter the gated spine (exit 4:
--reuse-dm <id> --wb-spec <path>).
"The DM is posted" is not a finish line. Do not report done or wait for a
human at these STOPs; the only finish line is verify-complete.rb exit 0.
- At the exit-4 handoff, PATCH the auto-built
<WORK>/wb-spec.json — do NOT
rewrite it. Edit ONLY the fields/tiles it names, keeping every other
element and every controlId exactly as built, then re-enter with
--reuse-dm <id> --wb-spec <WORK>/wb-spec.json. A from-scratch rewrite
drifts controlIds from control-scope.json and fails control-lint wholesale
(the #1 way this handoff spirals). For a master-level calc, prefer
--master-col over hand-editing the tile.
- Credentials: not on Claude Code → the orchestrator stops at step 0 telling
you to run
ruby scripts/setup.rb once — do that (bootstrap runs it
--from-env when the vars are exported) rather than working around auth.
Model fit & vision requirements — refs/model-fit.md. Pixel-fidelity
claims require image input; very-large workbooks on non-top-tier models
require asking the user once before proceeding. A text-only agent must record
the visual verdict as not-executable (named degradation), never pass. Read
refs/model-fit.md before Phase 0 on any multi-dashboard workbook.
Convert a Tableau datasource into a Sigma data model, then build a Sigma
workbook that mirrors the Tableau dashboard layout as closely as possible.
🚧 GATE convention. A step marked 🚧 GATE is backed by a script that
fails the run if the step was skipped — not advisory. Every 🚧 GATE requires
an artifact (a file on disk) and a downstream check that refuses to
proceed without it. Current gates: Phase 1d source dashboard-read
(png-read.json → assert-dashboard-read.rb, re-enforced inside
build-charts-from-signals.rb), Phase 6 source-parity and Phase 6f
visual render (assert-phase6-ran.rb).
Read ALL of the following before replying or taking any action. Do not make assumptions about skill conventions, prompts, or global instructions — read the files.
refs/operating-contract.md — READ FIRST. The non-negotiable guardrails: obtain the source render + calcs, build from the source's own logic (not SUM(col)), render + value-check EVERY page against the source, never ship empty or waive silently, don't spin.refs/composition-recipe.md — composition pass (hero/panels/carded KPIs + brand-from-source), value-fidelity rules, controls/params rebuild, spec/API gotchas.refs/column-gotchas.md — column naming rules and special-character landminesrefs/data-model-spec.md — data model JSON schema, element format, relationship formatrefs/workbook-layout.md — Ruby layout generation (mandatory), multi-series chart patternsrefs/story-points.md — Tableau stories → one Sigma page per story point (build-story-pages.rb)refs/blending.md — data-blend detection + routing decision treerefs/window-functions.md — Tableau window/table calcs → Sigma-native window mathrefs/coverage-matrix.md — converter-wide Tableau→Sigma coverage matrix (static companion to scan-workbook-gaps.rb)refs/source-anchors.md — the measured value bar: source-anchors.json schema, canonicalization, worked examplerefs/orchestration.md — builder/verifier split, fresh contexts, converter backend, the still-manual listrefs/performance.md — performance guidancerefs/script-map.md — the full per-script detail catalog (flags, artifacts, exit codes)
For canonical workbook spec shape (element kinds, controls, formulas,
formatting), defer to the companion sigma-workbooks skill (in the
sigma-authoring plugin, same marketplace — install it alongside; this
converter assumes it). This skill restates only Tableau-conversion-specific
patterns.
Step −1 — Mission intake (MANDATORY — before anything else)
Restate the user's request as <workdir>/mission.json (schema + rules in
MIGRATION_REQUEST.md): source, Sigma connection, destination, landing
target, workbook scope — each marked stated or inferred. Any inferred
field → STOP and confirm with the user before building. Field evidence: every
scope decision an agent invented has invalidated the entire run.
MIGRATION_REQUEST.md also carries the copy-paste kickoff template.
Step 0 — Bootstrap + doctor (MANDATORY — the orchestrator gates on it)
Run the bootstrap FIRST (STEP 0 banner above). It ends with the doctor, which
writes doctor.json (to ~/.sigma-migration/ and <WORK>/ with --workdir)
plus the bootstrap.json sentinel. migrate-tableau.rb refuses to start until
a passing doctor.json + sentinel exist — a broken environment stops here
with the exact fix instead of the run improvising around a missing runtime.
Re-verify anytime with bash scripts/doctor.sh --workdir <WORK> (PowerShell:
scripts\doctor.ps1 -WorkDir <WORK>). If the gate cannot pass in your
environment (e.g. a sandbox), waive it explicitly and name it in your report:
migrate-tableau.rb … --skip-doctor-gate "<reason>".
Step 0.1 — Front door: resolve the connection once (scripts/intake.rb)
Resolve the Sigma warehouse connection a SINGLE time so no phase free-searches
/v2/connections (the token sink):
ruby scripts/intake.rb --workdir <WORK> --tool tableau-to-sigma --mode live \
[--connection <id>] [--name <connection-name-substring>] [--source "<workbook name>"]
Credentials are shell-neutral. The Ruby/Python scripts mint Sigma tokens
themselves from SIGMA_CLIENT_ID/SIGMA_CLIENT_SECRET (env or
~/.sigma-migration/env) and auto-refresh — no eval step, any shell. For a
hand-driven curl, mint explicitly: python scripts/get_token.py --workdir <WORK> writes <WORK>/auth.json (0600), read automatically by the scripts.
(eval "$(scripts/get-token.sh)" still works in bash.)
It caches <WORK>/connection.json (the orchestrator reads it when
--connection is omitted — point --out at the same <WORK>) and writes
intake.json. Precedence: explicit --connection → cached →
SIGMA_CONNECTION_ID env → list once; with multiple connections it asks — it
never guesses. Many connections? Pass --rank-workbook-id <LUID>: intake
fingerprints the workbook's warehouse (type+host, no .twb download) and a
unique match auto-resolves, else it writes ranked connection-candidates.json
and asks (scripts/rank-connections.rb standalone; --rank-twb <path>
db-name tie-break). Never grep the .twb by hand to guess among connections.
One command (orchestrated path)
⏱ Invocation rule (G2, field-caught): a full pass runs 5–20+ minutes.
Launch it in the background (log you poll) or with a tool timeout ≥ 20
minutes — the default 2-minute foreground Bash limit WILL kill the pass
(exit 143). Check exit codes with portable $? (not bash-only PIPESTATUS).
ruby scripts/migrate-tableau.rb \
--workbook "<name>" --connection <SIGMA_CONNECTION_ID> --folder <SIGMA_FOLDER_ID> \
[--db <DB> --schema <SCHEMA>] [--name '<prefix>'] [--row-scale 1.5] \
[--reuse-dm [ID]] [--force] [--yes]
ruby scripts/migrate-tableau.rb --workbook "<name>" \
--finalize --actuals <workdir>/parity-actuals.json [--allow-missing-tiles N]
--db/--schema (always together) — there is NO default database. The
orchestrator resolves: your flags → the landing manifest → the workbook's own
<connection dbname=/schema=> attributes — and hard-stops asking when
none resolve. It never guesses: a fabricated pair 404s every table at DM POST.
🚧 Expect PASS 1 to hard-stop once for the Phase 1d dashboard-read. The
orchestrator fetches view CSVs but cannot read the source dashboard PNG —
that's your job. On a cold run it aborts before posting the DM with
"Phase 1d dashboard-read gate": fetch the dashboard PNG
(mcp__tableau__get-view-image, solo), Read it, write <workdir>/png-read.json
(every tile with its Sigma kind, + text_elements, filter_shelf — schema
in refs/phase-1-discover.md), then re-run the same PASS 1 command
(discovery artifacts are reused). PNG genuinely unavailable →
--skip-dashboard-read "<reason>".
Converter backend — LOCAL by default, zero config, never upload customer
data silently. A prebuilt converter ships at converter/tableau.mjs (pure
function, runs via node, no network); a dev build wins when configured
(TABLEAU_MCP_BUILD, SIGMA_DATA_MODEL_MCP, scripts/dev/fetch-converter.sh;
refresh via scripts/dev/vendor-converter.sh, pinned in
converter/PROVENANCE.json). The hosted converter uploads the .twb and
is used only with explicit --converter hosted or
SIGMA_CONVERTER_ALLOW_HOSTED=1. No converter at all? Re-enter the GATED
spine — never hand-POST raw specs: author dm-spec.json (see
sigma-data-models) + wb-spec.json (see sigma-workbooks; DM refs as
"__DM_ID__" / "__DM_ELEMENT__:<Name>", fact = "__DM_ELEMENT__:__FACT__")
and re-run with --dm-spec <path> --wb-spec <path> (or --reuse-dm <id> --wb-spec <path> when the DM is already posted). Full detail:
refs/orchestration.md §Converter backend.
Parity is EXACT for warehouse-backed migrations — never blame "drift."
Sigma queries the same warehouse Tableau reads; a value gap is a real bug
(missing view filter / NULL bucket / ungrouped table / wrong aggregate), not
freshness. The drift-tolerance path (extract-mode) is only for
hasExtracts=true workbooks and must be explicitly flagged. Do not declare
GREEN while a table renders base-row-count detail or a NULL-dominant bucket —
run scripts/lib/preflight_lint.rb on the spec first.
Chains the scripted spine (discover → scan-workbook-gaps → discover-columns →
find-or-pick-dm → DM build/validate/post → build-charts-from-signals → workbook
post-and-readback → build-dashboard-layout + put-layout → phase6-parity →
cleanup-orphan-workbooks + assert-phase6-ran) and STOPS with exact instructions
wherever agent judgment is genuinely required. Phase 1 is interleaved
(Tableau-side + Sigma-side lanes join before anything consumes discovery
output); a PHASE TIMINGS line prints at every terminal exit. Exit codes:
10 = OPEN QUESTIONS (re-run with --yes/--answers), 11 = ❌-unhandled
gap-scan features (gap-scout or --force), 12 = pass 1 done, parity PENDING
(collect mcp-v2 actuals, then --finalize), 16 = pass 1 built + POSTed but
<workdir>/manual-residues.json carries unbuilt window/table-calc residue(s)
a dashboard tile plots — build each from the printed checklist, set
status:"built", then --finalize (the final gate refuses GREEN on unbuilt
residues, exit 22; waiver --accept-manual-residues "<calc,...>",
budget-counted), 4 = DM posted but the workbook layer needs an agent-authored
spec — re-enter with --reuse-dm <id> --wb-spec <path> (never hand-POST),
3 = a gate failed, 14 = migration GREEN + Phase E proposals pending,
0 = ALL gates green (only reachable via --finalize).
DM-reuse is reuse-first (auto-reuses a covering DM, skipping the POST);
--reuse-dm <id> pins one, --skip-reuse-scan forces build-new. Optional
--enhance [--enhance-accept <ids|all-low-risk>] runs Phase E after all gates
are green. Still manual by design (pivot-grid parity actuals,
empty-view-CSV recovery, --master-col overrides, gap-scout escalations,
DM-reuse shape preflight): refs/orchestration.md §Still manual.
Scripts
migrate-tableau.rb composes everything (STEP 1) — the only entry point
you run cold. Invoke another script directly only when an orchestrator STOP
tells you to. One line each below; the FULL contract of every script (flags,
artifacts, exit codes, field rationale) is in refs/script-map.md — open
it when a STOP routes you to a script.
| Script | One line |
|---|
migrate-tableau.rb | The one command — chains the whole gated spine; stops with exact instructions |
verify-complete.rb | The single offline "are we done?" check — ✅ DONE only on gate-green; re-derives the verdict ledger (exit 6 on contradiction) |
lib/offramp.rb + offramps.jsonl | Observability trail of every golden-path exit |
setup.rb / setup-tableau.rb | One-time Sigma / Tableau credential setup (--from-env = non-interactive; bootstrap runs them) |
get-token.sh / get_token.py | Sigma token mint (bash / shell-neutral twin → <WORK>/auth.json) |
get-tableau-token.sh / get-tableau-token.py | Tableau PAT signin (bash / shell-neutral twin) |
bootstrap.sh / bootstrap.ps1 | Step-0 environment bootstrap → doctor-green + sentinel (--check dry run) |
doctor.sh / doctor.ps1 | Env check; writes the doctor.json fingerprint the orchestrator gates on |
assert-doctor-ran.rb | 🚧 GATE — refuse to run without passing doctor.json + bootstrap sentinel |
assert-wb-refs-resolve.rb | 🚧 GATE — every workbook [Element/Column] ref must exist in the live DM (waive --skip-ref-check "<reason>") |
tableau-discover.rb | PAT-mode Phase 1 discovery in one pooled CLI (13.7–18.9s on the 7-view reference) |
resolve-project.rb | Phase 1a numeric-URL resolver (⛔ no guessing; exit 2 = STOP and ask) |
scan-workbook-gaps.rb | Phase 0a (mandatory) gap scan → gaps.json / gaps-report.md + blend-plan.json |
gap-scout.md | Phase 0a-scout subagent protocol for ❌ Unhandled gaps |
validate-sigma-formula.rb / scout-validate-and-persist.rb | Scout primitives: probe a candidate formula; persist learned rules / escalations |
escalate-gap.py | Opt-in issue filer (dry-run by default; --yes to file) |
learned-rules.rb | Merges learned/starter-rules.yaml + ~/.tableau-to-sigma/learned-rules.yaml |
parse-twb-layout.rb | .twb → per-dashboard zone list + *-meta.json (+ story-plan.json for stories) |
build-charts-from-signals.rb | Zones + CSVs + master map → Sigma chart specs; writes coverage.json; 🚧 requires png-read.json |
extract-custom-sql.rb / resolve-published-ds.rb / hydrate-custom-sql.rb | Custom-SQL + published-DS (sqlproxy) resolution/hydration — never a phantom table |
lib/tableau_rest.rb | Tableau REST wrapper |
estimate-cost.rb | Phase 0c scope + token-cost estimate → cost-estimate.json + run-state ack |
fetch-view-data.rb | View CSVs → signals manifest |
discover-warehouse-columns.rb / probe-custom-sql-columns.rb | Phase 2 real column ids; 404 → catalog sync, then Custom-SQL probe |
find-prior-cache.rb | Reuse cached discovery artifacts from prior runs |
remap-wb-spec-to-dm-ids.rb | Re-point a cached wb-spec at a re-POSTed DM's new ids |
extract-calc-fields.rb | Phase 1e calc fields (+ formulas) → calc-fields.json |
validate-spec.rb | DM/workbook spec validator (--type, --dm-context) |
post-and-readback.rb | POST + GET-back a spec; column-type guard, layout+control lint, same-workbook PUT discipline (exit 7 unless STOP-authorized / --allow-manual-spec) |
lib/preflight_lint.rb | MANDATORY pre-POST lint (T1/T2 grouping, C1–C3 control shapes) |
put-layout.rb | Apply layout XML to an existing workbook |
auto-parity-plan.rb / verify-parity.rb | Phase 6a plan + 6c diff (--extract-mode only for hasExtracts=true) |
probe-equivalence.rb | MANDATORY proof for any structural edit (join drop / collapse / filter rewrite) → semantic-edits.json; mismatch = FATAL; --withdraw for unapplied edits |
derive-ground-truth.rb → run-ground-truth.rb → verify-ground-truth.rb | Phase 6 tile-grain warehouse oracle → ground-truth-plan.json + per-tile numeric_parity |
verify-anchors.rb | Phase 6 measured value bar → anchors-verdict.json |
assert-phase6-ran.rb | The conversion hard gate — see the gate table below |
fidelity-loop.rb | Phase 5g RCF mechanics (init/render/record/apply-patch/resolve/status) → fidelity-ledger.json |
assert-run-state.rb | Phase-chain ledger audit (run-state.json; --skip-run-state "<reason>") |
probe-controls.rb | Runtime control flip test (gate 7b) → probe-controls/probe-results.json |
cleanup-orphan-workbooks.rb | Delete spec-iteration orphans; writes cleanup-marker.json |
build-dashboard-layout.rb | MANDATORY Phase 5d layout XML from zones + wb-ids.json; emits layout-census.json |
build-story-pages.rb | Story workbooks: one Sigma page per story point |
export-chart-png.rb | Phase 6d drill-down only — the primary gate is full-dashboard vs full-dashboard |
pick-destination.rb | Phase 0b destination list/create |
find-or-pick-dm.rb / inspect-dm-shape.rb | Phase 1.5 reuse scan → dm-match.json; 1.5b shape preflight → dm-denorm-plan.json |
scan-customer-style.rb | Phase 0c house-style signals |
dev/phase-timer.sh | Dev/profiling only — never in customer conversions |
lib/layout.rb | Layout-XML helpers |
enhance-scan.rb / enhance-apply.rb | Phase E (opt-in) scan + accept-only clone-first apply |
The final gate (assert-phase6-ran.rb) — exit-code table (load-bearing)
Exits 0 only when ALL pass. Full prose per gate: refs/script-map.md +
refs/phase-6-parity.md.
| Exit | Gate | Fails when / escape |
|---|
| 1 | parity sentinel | parity-final.json missing |
| 2 | parity | FAIL / extract-mode-without-flag / charts_total==0 unbacked |
| 4 | orphans | uncleaned posted workbooks (cleanup-orphan-workbooks.rb) |
| 5 | live columns | any live type=error column |
| 6 | layout | no non-empty top-level layout applied |
| 7 | tile census | unexplained missing zones / --allow-missing-tiles N |
| 8 | layout lint | lib/layout_lint.rb / --skip-layout-lint |
| 9 | control lint | lib/control_lint.rb, control-scope.json / --skip-control-lint |
| 10 | visual render | no sigma-render.png or screenshots/_manifest.json / --skip-visual-gate "<reason>" |
| 13 | visual verdict | unrecorded or self-attested pass — needs sha-bound blind_grade (PR-9) or recorded --no-vision-waiver / --skip-visual-comparison "<reason>" |
| 14 | layout fill | layout-census.json dropped tile or grid_fill_pct < --min-grid-fill / --skip-layout-fill "<reason>" |
| 15 | 8d RCF ledger | unresolved spec-fixable deltas (--require-fidelity-ledger, default-on at --finalize); --accept-residuals id,id — data-class NEVER |
| 18 | 13 anchors | source-anchors.json <5 anchors or failing anchors-verdict.json; also --skip-parity-gate without passing anchors / --skip-anchors-gate "<reason>" |
| 19 | waiver budget | >2 quality waivers → YELLOW cap; no escape flag |
| 20 | 14 visual similarity | visual-similarity.py floor fail / --skip-visual-similarity "<reason>" |
| 21 | 7b control flip | control doesn't filter live (probe-controls.rb), or no evidence on an enforced run / --skip-flip-test "<reason>" → --skip-control-flip (budget-counted) |
| 22 | 15 manual residues | manual-residues.json still unbuilt / --accept-manual-residues "<calc,...>" (budget-counted) |
| 23 | 16 join cardinality | join-plan.json unprobed / non-unique Lookup target (probe-join-keys.rb, --resolve <i> --how <preaggregated|waived> --reason); no skip flag |
| 24 | 17 LOD ledger | lod-audit.json suspect-alias / silently-dropped unresolved (audit-lod-calcs.rb); no skip flag |
| 25 | 18 ground truth | displayed tile with no numeric_parity match and no coverage_waivers entry; diverge/conflict NEVER waivable; no skip flag |
| 26 | 19 agg semantics | agg-semantics.json unresolved additive-over-preagg / countd-as-sum / preagg-ratio (audit-agg-semantics.rb --how <reaggregated|n/a|faithful-to-source>); no skip flag |
| 27 | 20 semantic edits | semantic-edits.json unproven or match:false (probe-equivalence.rb); NO waiver — revert or redesign |
| 28 | 21 chart-kind parity | live element family ≠ verified png-read.json kind; substitutions recorded as kind_waivers at read time |
| 30 | 4b layout ran | run-state shows the layout phase never entered |
| 31 | 7c controls census | source control never built, undeclared in control-scope.json and unnamed in controls-waivers.json; no skip flag |
On the all-embedded (charts_total==0) ANCHORS-ORACLE path, substitution also
requires per-displayed-tile anchor coverage (anchor_coverage.covered == displayed, or coverage_waivers naming each uncovered tile). Subagent flows
MUST call this gate as their final step.
Prerequisites
⛔ NEVER hand-roll a signin curl — for Sigma OR Tableau. Managed
machines' permission classifiers block secret-bearing raw curl, and
curl -sf hides the failure (a field session lost ~15 minutes to it). The
scripts are the sanctioned AND classifier-friendly route: setup.rb /
setup-tableau.rb once, in-process token minting everywhere, and
get_token.py / get-tableau-token.py for hand-driven REST.
Sigma credentials
ruby scripts/setup.rb once (bootstrap runs it --from-env when the vars are
exported). Writes ~/.claude/settings.json (Claude Code auto-loads) and
~/.sigma-migration/env (neutral, auto-sourced by the scripts under any
agent). Required: SIGMA_BASE_URL, SIGMA_CLIENT_ID, SIGMA_CLIENT_SECRET.
Tokens live ~1h — scripts auto-refresh; for hand-driven calls,
python scripts/get_token.py --workdir <WORK> (any shell). Shell footguns
(subshell eval, inline-python quoting): refs/troubleshooting.md §Shell.
Tableau access — two modes
Prefer the API/PAT path — measured 61.8s serial → 13.7–18.9s pooled on
the 7-view reference; the MCP is the no-PAT fallback only (each MCP fetch
is a separate agent turn).
| Mode | When | Setup |
|---|
| PAT (REST) — preferred | A Tableau PAT is available; only path to .twb content | ruby scripts/setup-tableau.rb once |
| MCP — fallback | No PAT, mcp__tableau__* tools loaded | None — host handles auth |
PAT mode: migrate-tableau.rb needs no token step (in-process mint, works
in PowerShell). Hand-driven only: eval "$(scripts/get-tableau-token.sh)"
(bash) or python scripts/get-tableau-token.py --print-token, then
ruby scripts/tableau-discover.rb --workbook-id <luid> --out <WORK> [--pool N]
→ same artifacts as MCP-driven Phase 1 in one run (workbook JSON, .twb,
ds-metadata.json, graphql-fields.json, views/*.csv, dashboard PNG,
timings.json). --datasource-name/--datasource-luid optional (auto-detect;
--no-auto-ds disables; LUID must be the full UUID). Pool mechanics + why 5:
refs/tableau-rest.md §fetch pool. Endpoint inventory: refs/tableau-rest.md.
One signin attempt only. Tableau Cloud invalidates a PAT after 4 failed
signins. get-tableau-token.sh runs exactly once; never wrap it in a retry loop.
The workflow (progressive disclosure)
This spine is the map: every phase, its one command, its gate. The how /
why / gotchas live in refs/phase-*.md — open the ref for the phase you're
on, when you're on it (the flat version of this skill made agents drop steps).
| # | Phase | One command / action | Gate → artifact | Detail |
|---|
| 0 | Preflight + intake | bootstrap.sh; doctor.sh; intake.rb | bootstrap.json + doctor.json + connection.json | (spine ↑) |
| 0a | Gap scan (mandatory) | scan-workbook-gaps.rb | gaps.json — ❌ features → scout or --force | refs/phase-0-scope.md |
| 0b | Destination + mode (ask) | pick-destination.rb | folder id + conversion mode | refs/phase-0-scope.md |
| 0c | Scope/cost sign-off | estimate-cost.rb --workdir | cost-estimate.json + run-state ack (cost_estimate_acknowledged) | refs/phase-0-scope.md |
| 1 | Discover the source | tableau-discover.rb (PAT) or MCP | get-workbook.json, views/*.csv, .twb | refs/phase-1-discover.md |
| 1d | 🚧 Dashboard read + anchors | get-view-image (solo) → Read → write png-read.json + source-anchors.json | 🚧 png-read.json (assert-dashboard-read.rb) + ≥5 anchors (final gate exit 18) | refs/phase-1-discover.md, refs/source-anchors.md |
| 1.5 | Reuse an existing DM | find-or-pick-dm.rb | dm-match.json (reuse-first) | refs/phase-1_5-dm-reuse.md |
| 2 | Warehouse column names | discover-warehouse-columns.rb | real column ids | refs/phase-2-columns-filters.md |
| 2.5 | View-level filters (mandatory) | detect from CSV distinct values | filters applied at the right grain | refs/phase-2-columns-filters.md |
| 3 | Build the DM spec | author → validate-spec.rb --type datamodel | clean dm-spec.json + join-plan.json probed (exit 23) + semantic-edits.json proven (exit 27) | refs/phase-3-datamodel.md |
| 4 | POST the DM + read back | post-and-readback.rb --type datamodel | dm-ids.json (server ids) | refs/phase-4-post-dm.md |
| 5 | Build the workbook | build-charts-from-signals.rb → post-and-readback → build-dashboard-layout.rb → put-layout.rb | preflight_lint clean; layout is the LAST write; lod-audit.json (exit 24) + agg-semantics.json (exit 26) resolved | refs/phase-5-workbook.md |
| 6 | 🚧 Parity + anchors + ground truth + visual | phase6-parity.rb; verify-anchors.rb; ground-truth trio; then the gate sequence | 🚧 parity-final.json PASS + anchors-verdict.json + per-tile numeric_parity (exit 25) + recorded visual verdict + full run-state.json | refs/phase-6-parity.md, refs/source-anchors.md, refs/ground-truth-oracle.md |
| 5g | RCF fidelity loop | fidelity-loop.rb render → compare → fix until clean | 🚧 (default-on) fidelity-ledger.json no unresolved spec-fixable deltas (gate 8d) | refs/phase-5g-rcf.md |
| E | Enhance (opt-in) | enhance-scan.rb → enhance-apply.rb | cloned "— Enhanced" workbook | refs/phase-e-enhance.md |
| — | Security RLS/CLS | detect always; apply opt-in | apply_sigma_rls.py | refs/security-rls.md |
| — | Telemetry (final) | report-telemetry.py | telemetry-sent.json | (spine ↓) |
The orchestrated path runs 0a–6 for you and stops with exact instructions
where judgment is required. Reach for the per-phase refs when you drive a
phase by hand or need to understand why it stopped.
Phase stanzas (open the ref for the one you're on — READ IT AT THAT MOMENT)
Phase 0 — scope: gap scan, destination, mode, cost — refs/phase-0-scope.md
scan-workbook-gaps.rb on the .twb before anything else (❌-unhandled →
gap-scout or --force); then pick-destination.rb, conversion mode, cost
estimate. Then the model-fit checkpoint (refs/model-fit.md §3):
large/very-large (or >1 dashboard / >30 zones / any ❌ rows / >50 calcs) on a
non-top-tier model → ask the user once; never silently proceed on very-large.
Phase 1 — discover the source — refs/phase-1-discover.md
PAT mode: tableau-discover.rb (one pooled run). MCP is the no-PAT fallback.
Calc fields via extract-calc-fields.rb, Custom SQL via extract-custom-sql.rb.
🚧 GATE — Phase 1a: ANY numeric Tableau URL (project or workbook) MUST go
through resolve-project.rb first. The number is a vizportal URL id the
REST API cannot resolve. Exit 0 → migrate exactly what it lists. Exit 2 →
STOP and ask the user with the printed candidates — never guess (a wrong
guess field-cost 6 hours on the wrong project). /views/<slug>/<view> share
links: paste the whole URL as --workbook "<url>" — never hand the slug to a
name lookup. Full rationale + query shapes: refs/phase-1-discover.md,
refs/tableau-rest.md.
🚧 GATE — Phase 1d dashboard-read + source-value anchors. Read the source
dashboard PNG (solo get-view-image) → write png-read.json (EVERY tile +
kind; build-charts-from-signals.rb refuses without it) AND transcribe the
printed numbers into source-anchors.json exactly as printed ("12,345B",
never 12345): every KPI, top-3 of every ranked list, one bucket value per
chart, ≥5 total, anchor EVERY tile (a tile with no anchorable value gets a
coverage_waivers entry at transcription time). Phase 6 verifies each anchor
against the LIVE exports (verify-anchors.rb); the final gate fails (exit 18)
on missing/unverified anchors whenever a source PNG exists. Two field
migrations shipped wrong numbers behind passing visual verdicts — this is the
fix. Schema + rules + worked example: refs/source-anchors.md. Escapes:
--skip-dashboard-read "<reason>", --skip-anchors-gate "<reason>" (budget-counted).
Phase 1.5 — reuse an existing DM (do this first) — refs/phase-1_5-dm-reuse.md
find-or-pick-dm.rb scores existing DMs (reuse-first skips Phases 2–3);
differently-shaped reuse → run 1.5b inspect-dm-shape.rb.
Phase 2 + 2.5 — warehouse columns + view filters — refs/phase-2-columns-filters.md
discover-warehouse-columns.rb; on 404 force a catalog sync
(POST /v2/connections/{id}/sync) and retry before the Custom-SQL probe. Then
detect view-level filters (mandatory) by diffing CSV distinct values, and
apply at the correct grain. Relative-date → native ROLLING mapping: same ref.
Phase 3 — build the DM spec — refs/phase-3-datamodel.md
Author the spec (warehouse-table element for plain columns, sql elements for
window/LOD calcs), translate calc fields, validate-spec.rb --type datamodel
before POST. Schema: refs/data-model-spec.md. Structural edits need
probe-equivalence.rb proof; Lookups need join-plan.json probes.
Phase 4 — POST the DM + read back — refs/phase-4-post-dm.md
post-and-readback.rb --type datamodel POSTs then GETs back the server ids the
workbook binds to (column-type guard aborts on type=error).
Phase 5 — build the Sigma workbook — refs/phase-5-workbook.md
build-charts-from-signals.rb (🚧 requires png-read.json) →
preflight_lint.rb (mandatory) → post-and-readback.rb --type workbook →
build-dashboard-layout.rb → put-layout.rb. Layout is the LAST write — a
bare spec PUT wipes it. Then compile-check every element.
Phase 6 — 🚧 parity + anchors + visual verification (hard-gated) — refs/phase-6-parity.md
phase6-parity.rb diffs Sigma actuals vs Tableau (EXACT for warehouse-backed;
--extract-mode only for hasExtracts=true). Phase 6f visual is mandatory:
render the full Sigma page and Read it against the source PNG. Your own read
is the fix loop, not the verdict — when you believe it matches, spawn a
FRESH context-free blind grader (prompt = refs/blind-grader-brief.md with
exactly four parameters: source PNG, render PNG, rubric path, output path;
pass NOTHING else), then
record-visual-check.rb --blind-grade <WORK>/blind-grade.json — a pass without
a passing blind grade is REFUSED (gate 8b re-verifies hash-bound). FAIL →
record divergent, fix, re-render, re-grade fresh. No vision-capable subagent
→ --no-vision-waiver "<reason>" (budget-counted). Full protocol +
no-standalone-views (anchors+warehouse oracle) + similarity floor:
refs/phase-6-parity.md. Finish with the gate sequence:
ruby scripts/assert-dashboard-read.rb --workdir <WORK>
ruby scripts/verify-anchors.rb --workdir <WORK> --workbook-id <wb>
ruby scripts/assert-run-state.rb --workdir <WORK>
ruby scripts/assert-phase6-ran.rb --workdir <WORK> --workbook-id <wb>
Not done until assert-phase6-ran.rb exits 0 (stamps
phase6-success.json, run-scoped — quote its workbook id + run id verbatim),
then verify-complete.rb prints ✅ DONE.
Waiver discipline — waivers are for impossibilities, not obstacles. Every
--skip-* / --allow-* / --accept-* / --min-pass-rate <1 flag attests a
verification could NOT run — not a lever to turn a red gate green. The gate
stamps waivers + waiver_count into parity-final.json; more than 2
quality waivers caps the migration at YELLOW (exit 19; --skip-telemetry-gate
and the sanctioned builder→verifier --skip-visual-comparison are policy
exclusions), and data-class fidelity residuals can never be waived.
Reaching for a third waiver means: stop and fix.
Verdict model (PR-14) — GREEN / YELLOW / PARTIAL. Every gate run derives
degradation-ledger.json from the artifacts (lib/degradation_ledger.rb —
mechanical, never self-reported) and prints ONE verdict with the ledger
inline. GREEN requires an EMPTY ledger. Any non-scope-cut entry (or
exceeded budget) → YELLOW. Any scope cut caps at PARTIAL. Stamped into
phase6-success.json + parity-final.json; verify-complete.rb re-derives
offline and fails (exit 6) if your report contradicts it — quote the
ledger verbatim, never a verdict it doesn't support.
Phase 5g — RCF (render-compare-fix) fidelity loop — refs/phase-5g-rcf.md
fidelity-loop.rb render → Read vs source + score (refs/fidelity-rubric.md)
→ record each delta → fix from refs/fidelity-recipes.md + apply-patch
(single layout-preserving PUT) → loop until status is clean. DEFAULT-ON gate
8d (exit 15); --rcf-passes 0 opts out but records the named
--skip-fidelity-gate waiver (budget-counted). Layout build also emits
layout-arrangement.json (gate 8e, WARN this release, --require-arrangement
to enforce); gate 4b (exit 30) fails a run whose ledger shows layout never ran.
Phase E — enhance (opt-in) — refs/phase-e-enhance.md
After all gates green: enhance-scan.rb proposes; enhance-apply.rb applies
only accepted candidates to a cloned workbook (parity artifact never mutated).
Troubleshooting — refs/troubleshooting.md
Common failure modes + fixes (incl. shell footguns): refs/troubleshooting.md.
Security: Row- & Column-Level Security (RLS/CLS) — refs/security-rls.md
RLS/CLS is never silently dropped and never silently ported — the converter
only detects and reports (result.security[], USERNAME()/ISMEMBEROF/
USERATTRIBUTE → CurrentUserEmail()/CurrentUserInTeam/
CurrentUserAttributeText); this skill provisions + applies AFTER the model is
posted. Non-empty result.security → write security.json, summarize, ask
Port (recommended) / Customize / Skip, then python3 scripts/apply_sigma_rls.py --from-security security.json --dm-id <id> (plan by
default; --provision --apply to execute), assign membership. Skip is loud
— all rows visible to everyone; confirm first. Full flow: refs/security-rls.md.
Telemetry (after the final gate passes)
Tell the user this in the conversation before running anything:
"Migration complete. Before I wrap up, I'd like to send an anonymous usage ping so we can track which migration skills are being used. It records: tool name, your Sigma region, an anonymized org fingerprint (a hash of your client ID — not the credential itself), migration duration, and success. No workbook names, SQL, column names, or any customer data is included. See TELEMETRY.md for the exact payload. Just say 'skip' if you'd prefer not to send it."
If the user does not object, run:
python3 scripts/report-telemetry.py --tool tableau-to-sigma --duration <elapsed_seconds> --workdir <run-dir> [--mode live|file|both]
