| name | ts-convert-from-tableau |
| description | Convert or import a Tableau workbook (.twb or .twbx) into ThoughtSpot — parses TWB XML, generates table + model TMLs, validates and imports. Optionally migrates dashboards to liveboards with layout approximation. Direction is always Tableau → ThoughtSpot. Not for ThoughtSpot → Tableau or standalone TML exports. |
Tableau Workbook → ThoughtSpot
Converts a Tableau workbook into ThoughtSpot objects. Parses the TWB XML to extract
tables, columns, joins, and calculated fields, then generates Table TMLs and a Model
TML per datasource. Optionally converts Tableau dashboards into ThoughtSpot Liveboards
with approximate layout mapping.
Ask one question at a time for dependent decisions (where the next question depends on
the answer), waiting for each. But batch independent questions into a single
multi-question prompt to cut round-trips and keep the migration fast — e.g. mode + scope,
the count-column + bin-style + cohort-handling decisions, or theme + parameter-chips. See
Efficiency in Step 0.
References
Prerequisites
- ThoughtSpot profile configured — run
/ts-profile-thoughtspot if not
ts CLI installed: pip install -e tools/ts-cli- Tableau workbook file (
.twb or .twbx) accessible on disk
- Tableau profile configured (optional) — run
/ts-profile-tableau if migrating workbooks
with published datasources (sqlproxy). Not needed for workbooks with direct connections.
- The source tables and their data already exist in a warehouse, and a ThoughtSpot
connection exposes them. This skill creates ThoughtSpot logical objects (Table, Model,
cohorts, Liveboard) over existing physical tables — it does not create warehouse
tables or load/populate data. A ThoughtSpot table binds to a live connection that already
surfaces the physical table and its columns (see Step 4.5 /
thoughtspot-table-tml.md); if no
such connection/table exists, set that up first (the data pipeline is out of scope). The
skill may read the warehouse for confirmation (value formats, ranges, membership) — with
your authorization — but never loads or modifies data.
CLI-first rule — no inline Python for TML operations
Every ThoughtSpot API call, TML generation, formula translation, and model import in
this skill must go through a ts CLI command. Do not write inline Python scripts to
export/merge/import TML, iterate over formula failures, or assemble model JSON.
| Operation | Use this | NOT this |
|---|
| Translate formulas | ts tableau translate-formulas | Inline Python calling tableau_translate.py / the ts_cli/tableau/ package modules |
| Build/merge model + formulas | ts tableau build-model --existing-guid | Manual export → JSON merge → import loop |
| Create tables | ts tables create | Raw HTTP calls to the TML import endpoint |
| Export/import TML | ts tml export / ts tml import | Manual curl or HTTP library calls |
If a CLI command fails or produces wrong results, fix the CLI (tools/ts-cli/) and
re-run — do not work around it with manual scripting. The CLI encodes months of
edge-case fixes (parameter conflict renaming, cross-reference resolution, iterative
retry, double-aggregation detection) that inline scripts will miss.
Working principle — surface, recommend, resolve
Whenever the parse or generation hits a situation that has no clean 1:1 automatic
translation or needs a judgement call — e.g. a cross-datasource blend, a join key that
doesn't exist / spans two tables, a date stored as VARCHAR, bins (formula vs cohort),
an ambiguous count column, a manual group (cohort vs if/then), an untranslatable
formula, or a value-vs-data mismatch — do not silently drop it, guess, or merely
flag it. Instead:
- Surface it — tell the user plainly what was found and why it's not a straight
translation.
- Recommend — if there's a sound solution (or a small set of options), say which and why,
with the trade-offs.
- Resolve — with the user's go-ahead, do it (build the SQL view, prompt for the
value, retype the column, create the cohort, etc.). Only fall back to omit-and-flag when no
solution exists or the user declines.
Default to enabling the migration, not abandoning the hard parts. The per-step prompts and
checkpoints below are how this principle is applied in practice.
Read the actual calculation — never infer from the name. A worksheet called "Highest
Growth in past 5 years" tells you the intent, not the logic. Always inspect the real
Tableau definition — the table-calc type (pcdf, pctd, running_*), the filters
(Top-N, recent-N-years), the compute-using/partition, and the sort — and translate
that. (Example: that title is really "top 5 sectors by FDI % change over a 6-year window" —
a period comparison, not a raw growth of line.)
Placeholder charts when a full translation isn't possible. If a viz can't be fully
reproduced, don't silently omit it — build a placeholder: a TABLE with the columns you
can produce, and write a note in both the viz's answer.description and the Migration
Summary tab that the chart is partial and needs review. A visible, labelled stub the user
can finish beats a missing tile.
Step 0 — Overview
On skill invocation, display this plan before doing any work:
ts-convert-from-tableau — convert a Tableau workbook into ThoughtSpot TML objects,
with optional dashboard-to-liveboard migration.
Modes
A Audit — analyse a TWB file (or multiple files) and report migration coverage.
No ThoughtSpot auth required. No TMLs generated. Use this to assess feasibility
before committing to a migration.
M Migrate — full conversion: parse, generate TMLs, validate, and import.
Enter A / M:
Migrate scope (ask right after M — see Step 1.5)
When the user picks M, immediately ask what to migrate — this decides which steps run:
1 Models + Liveboards — full flow (tables/models, then dashboards → liveboards). (default)
2 Tables + Models only — build the data layer; skip dashboards/liveboards
(skip Steps 8–11; go model → Step 11.5 coverage → Step 12 report).
3 Liveboards only — the model(s) already exist in ThoughtSpot; skip table/model
creation (skip Steps 4–7.5) and build liveboards on a user-selected existing model
(see Step 1.5 model picker). Still parse the TWB for dashboards (Step 3).
4 Models only — tables already exist in ThoughtSpot; skip table creation but
build model(s) with formulas. Reuse existing table GUIDs (Step 4 E/G path).
(skip Steps 5a table TML generation, 8–11; run Steps 4→5b→5.5→6→7→7.5→11.5→12).
5 Tables only — generate and import table TMLs only; skip models and liveboards
(run Steps 4→4.5→5a→6→12). Useful when tables need to be created/updated before
a separate model migration pass.
Steps (Migrate mode)
- Authenticate to ThoughtSpot .......................... auto
1.5 Choose migration scope (1–5) and pace (F/C) ...... you choose
- Locate and extract the TWB file ...................... you provide path
- Parse TWB XML — extract tables, columns, joins,
calculated fields, blend relationships,
table-calc addressing ............................ auto
3.5 Resolve published datasources (sqlproxy → API) ... auto/you choose [scope 1,2,4,5]
3.6 Confirm joins (present/suggest/range join option) . you confirm [scope 1,2,4]
- Confirm source tables (reuse/GUID/create/search) ..... you choose [scope 1,2,4,5]
4.5 Select ThoughtSpot connection (create path only) .... you choose [scope 1,2,5]
- Generate TML files ................................. auto [scope 1,2,4,5]
5a Table TMLs (+ sql_view) ......................... auto [scope 1,2,5]
5b Model TML + formula translation ................. auto [scope 1,2,4]
5.5 Confirm Spotter (AI search) enablement (default Y) .. you choose [scope 1,2,4]
- Validate against ThoughtSpot (up to 10 fix cycles) .. auto [scope 1,2,4,5]
- Review checkpoint + two-phase import: you confirm [scope 1,2,4]
Phase 1: base model (tables, columns, joins, params — NO formulas)
Phase 2: add formulas (GUID-pinned update, iterative error recovery)
7.5 Confirm the model is correct (test in Search/Spotter) you confirm [scope 1,2,4]
- Migrate dashboards? + separate vs single-tabbed (2+) . you choose (skip → Step 12) [scope 1,3]
- Parse dashboard layout and map to grid ............... auto [scope 1,3]
9d. Orphan worksheets (not on a dashboard) — add as tiles? you choose [scope 1,3]
10c. Choose charting library (Legacy default / Muze) ........ you choose [scope 1,3]
- Generate liveboard TML (export model for params first) auto [scope 1,3]
10f. Add referenced parameters to the header? (default Y) . you choose [scope 1,3]
10g. Add a "Migration Summary" tab (migrated/decisions/omitted) auto [scope 1,3]
10.5 Pick a liveboard style (curated theme; default) ..... you choose [scope 1,3]
- Import liveboard ..................................... you confirm [scope 1,3]
11.5 Formula coverage answers (every formula testable) ... auto [scope 1,2,4]
- Migration report (outcomes + links + formula map) ... auto [scope 1,2,3,4,5]
12.5 Resume prompt — fix parked formulas? .............. you choose [scope 1,2,4; if parked]
Confirmation required: Steps 1.5, 3.6, 4.5, 5.5, 7, 7.5, 8, 9d, 11, 12.5
Auto-executed: Steps 1, 3, 5, 6, 9, 10, 11.5, 12
Scope 2 (Tables + Models) skips 8–11; runs 11.5 then 12.
Scope 3 (LB only) skips 4–7.5; runs 1.5a model picker then 8–12.
Scope 4 (Models only) skips 4.5, 5a, 8–11; runs 4(E/G), 5b, 6–7.5, 11.5, 12.
Scope 5 (Tables only) skips 5b, 5.5–7.5, 8–11.5; runs 4, 4.5, 5a, 6, 12.
Efficiency — keep the migration fast
The flow is interactive, but most of the wall-clock cost is avoidable. Apply these:
- Batch independent prompts. Use a single multi-question prompt for decisions that don't
depend on each other: mode + scope; the count-column + bin-style + cohort-handling
decisions; theme + parameter-chips. Only serialize genuinely dependent questions
(e.g. search-scope → connection name).
- Parse the TWB in one pass. Extract datasources, columns, calc fields, parameters,
dashboards, zones, and table-calc addressing in a single script — not one Bash call per
element.
- Read the model's real
obj_id once, up front (Step 10-pre) — exporting the model
once yields obj_id and parameters[].id and the resolved column names. This
prevents the slow build→import→fail→delete→re-import liveboard cycle (see the obj_id rule
in Step 7 / Step 10-pre).
- Don't fetch what you don't need. Skip
ts connections list / ts connections get
whenever the user names the connection or the tables are reused (Steps 4/4.5). Skip the
whole model/table layer entirely in scope 3 (LB only).
Steps (Audit mode)
A1. Locate and extract TWB file(s) ...................... you provide path(s)
A2. Parse TWB XML — same extraction as Step 3 .......... auto
A3. Classify formulas into translation tiers ............ auto
A4. Migration coverage report ........................... auto
No auth, no TML generation, no import. Supports multiple files in one run.
If Audit mode, proceed to Step A1. If Migrate mode, proceed to Step 1 (then Step 1.5 picks
the migration scope, which gates the later steps).
Step A1 — Locate TWB File(s) (Audit Mode)
Ask: "Provide the path to a .twb or .twbx file, or a directory containing multiple
workbooks."
If a directory is provided, find all .twb and .twbx files recursively. For each
.twbx, extract to a temp directory to access the inner .twb.
Save the list of TWB paths. Process each file through Steps A2–A4 independently.
Step A2 — Parse TWB XML (Audit Mode)
Run the same extraction as Step 3 (3a through 3e) on each TWB file. Do NOT skip any
datasource type. For extracts, resolve the underlying source (Step 3b) and report it as
migratable via that source; mark as "Extract — no underlying source" only when none
resolves.
Step A3 — Classify Formulas (Audit Mode)
MANDATORY (I7) — before classifying any calculated field as untranslatable, open
../../shared/mappings/tableau/tableau-formula-translation.md
and check its full function table and pass-through section. Do not decide from syntax alone.
Run the classifier — it shares the migrate-mode translation verdict, so the audit
cannot over- or under-promise coverage:
ts tableau classify-formulas --input /tmp/ts_tableau_mig/{workbook_name}_parsed.json --output /tmp/ts_tableau_mig/audit/{workbook_name}_classification.json
The classifier works per datasource — each datasource becomes its own model in
migration, and the same calc name can carry a different expression in each
datasource, so it must be tiered against its own. For a parsed-workbook input the
output is { "datasources": [ {"name", "formulas", "tier_counts", "translate_stats"}, … ], "tier_counts": <sum across datasources> }. Each formulas[] entry has tier, reason,
level, and complexity. Report per datasource (Step A4's per-datasource breakdown)
and use the top-level tier_counts for the workbook total. (Pass --datasource "<name>"
to limit to one; a bare-list input — e.g. Step 5b's translate-formulas file — instead
yields a flat {formulas, tier_counts, translate_stats}.)
Translatable tiers: native, lod, cumulative, moving, pass_through,
row_offset_native, parameter_ref. Untranslatable tiers: untranslatable,
row_offset_ambiguous, geospatial, circular, orphan, parameter_query.
The table below maps those tiers to the human-readable categories used in the report —
kept as reference/documentation now, not as executed classification logic:
| Tier | Description | Examples |
|---|
| Native / Set | Direct ThoughtSpot mapping exists | IF/THEN, IFNULL, DATEDIFF, LEFT, ABS, ROUND, IIF; bins (class='bin') → floor([x]/size)*size or BIN_BASED cohort; manual groups (class='categorical-bin', incl. fields named "… clusters") → GROUP_BASED cohort; Number of Records/row counts → count([column]) (prompt for the column; default the primary key); static sets (<group> with union/member) → GROUP_BASED column-set cohort — incl. ones anchored on a formula column, with a %null% member (via EQ {Null}), or an except member-list (via NE) (Phase 2a); Top-N/Bottom-N sets (function='end') → query set (cohort_type: ADVANCED, COLUMN_BASED) via a rank formula + parameter-filter formula (Phase 2b); condition-based sets (function='filter') → query set with aggregate condition formula (Phase 2c); member-list intersect → GROUP_BASED cohort of common members (Phase 2c); all-except-Top-N → query set with inverted rank filter (Phase 2c); computed set operations (intersect/except of mixed types) → multi-formula query set (Phase 2c) |
| LOD | LOD expression → group_aggregate() | {FIXED dim : SUM(col)}; TOTAL(SUM(x)) / percent-of-total → group_aggregate(..., {}, query_filters()) |
| Cumulative | Running calculation → cumulative_*() | RUNNING_SUM, RUNNING_AVG |
| Moving | Window table calc → moving_*() | WINDOW_SUM, WINDOW_AVG (when sort attr determinable) |
| Pass-through | Valid SQL but no native function → sql_*_aggregate_op() | Partitioned RANK, DENSE_RANK, WINDOW_* without sort context |
| Partial / Unmapped (sets) | Tableau set construct with no current ThoughtSpot equivalent — logged as deferred, never mis-translated | set controls (level-members only, no fixed members) → no set object, surface as a liveboard filter; set actions (<action>) → no equivalent |
| Row-offset (native) | Table calc with recoverable intent → native TS function | INDEX() <= N (Top-N filter → rank() + query set); INDEX() display → rank(); LOOKUP(-N) → moving_sum(N,-N); LOOKUP(+N) → moving_sum(-N,N); LOOKUP(agg, FIRST()) → first_value(); LOOKUP(agg, LAST()) → last_value(); bare FIRST()/LAST() standalone → omit + log (returns offset, not value) — all native mappings use native TS functions, work with all column types |
| Row-offset (pass-through) | SIZE() only → answer-level sql_int_aggregate_op("COUNT(*) OVER()") | SIZE() — only row-offset that still requires SQL pass-through ⚑ flag PT1 |
| Untranslatable | No ThoughtSpot equivalent — will be omitted | INDEX/LOOKUP/FIRST/LAST/SIZE when addressing is ambiguous (CellInPane, multi-dim Table, or no shelf sort); PREVIOUS_VALUE (true recursion — not the string-aggregation technique); true k-means clustering (the analytics-engine "Clusters" calc — not categorical-bin); geospatial — full 13-function set (MAKEPOINT, MAKELINE, BUFFER, OUTLINE, DISTANCE, AREA, LENGTH, INTERSECTS, SHAPETYPE, DIFFERENCE, INTERSECTION, SYMDIFFERENCE, VALIDATE) — decompose MAKEPOINT lat/lon args to individual attribute columns, omit the spatial formula (see tableau-formula-translation.md Geospatial Policy); embedded-RLS user attributes (USERATTRIBUTE, USERATTRIBUTEINCLUDES) — rejected at translate time, see BL-071 |
| Parameter ref (auto) | References a Tableau parameter with static list/range — parameter auto-created in model | [Parameters].[Currency] where Currency has <member> values |
| Parameter ref (query) | References a Tableau parameter with SQL-lookup list — queryable at migration time | SQL-populated parameter lists (needs connection) |
Step A4 — Migration Coverage Report (Audit Mode)
For each TWB file, produce a coverage report. If multiple files were audited, also
produce a combined summary at the end.
Source the numbers from classification.json (Step A3's ts tableau classify-formulas
output) — do not hand-tally tiers. The output is per datasource: iterate
classification.json's datasources[] for the per-datasource breakdown (each carries its
own formulas[], tier_counts, and translate_stats — where total == translated + skipped for that model), and use the top-level tier_counts for the workbook total. The
per-formula rows (Row-offset detail, Excluded Formulas, "Needing Review") come from each
datasources[].formulas[] entry's tier/level/complexity/reason fields.
Per-file report:
Audit: {workbook_name}
══════════════════════════════════════════════════════
Datasources: {N} total
Live: {N}
Extract: {N} (skipped in migration)
Published (sqlproxy): {N}
Physical tables: {N}
Custom SQL relations: {N} → will generate sql_view TMLs
Joins: {N}
Calculated fields: {N} total
┌──────────────────────────────────────────────────────┐
│ Tier Count % Examples │
├──────────────────────────────────────────────────────┤
│ Native {N} {%} IF, DATEDIFF │
│ LOD → group_agg {N} {%} {FIXED ...} │
│ Cumulative {N} {%} RUNNING_SUM │
│ Moving {N} {%} WINDOW_SUM │
│ Pass-through {N} {%} DENSE_RANK │
│ Row-offset (native) {N} {%} LAG→moving_sum│
│ Row-offset (pass-thru) {N} {%} SIZE→COUNT(*) │
│ Parameter ref (auto) {N} {%} static list │
│ Parameter ref (query) {N} {%} SQL lookup │
│ Geospatial (omit+log) {N} {%} MAKEPOINT │
│ Untranslatable {N} {%} INDEX(ambig) │
└──────────────────────────────────────────────────────┘
Cross-reference depth (formula-to-formula dependencies):
┌──────────────────────────────────────────────────────┐
│ Depth Count % Note │
├──────────────────────────────────────────────────────┤
│ Level 0 (self-contained) {N} {%} Translate directly │
│ Level 1 (refs Level 0) {N} {%} Inline from Level 0 │
│ Level 2+ (deep chains) {N} {%} Multi-level inlining │
│ Circular dependencies {N} {%} Cannot resolve │
└──────────────────────────────────────────────────────┘
Formula complexity (effort estimate):
┌──────────────────────────────────────────────────────┐
│ Complexity Count % Criteria │
├──────────────────────────────────────────────────────┤
│ Simple (1–2) {N} {%} ≤1 function, no cross-refs, no nesting │
│ Medium (3–5) {N} {%} 2–3 functions or 1 cross-ref or 1 nesting level │
│ Complex (6+) {N} {%} 4+ functions, 2+ cross-refs, or deep nesting │
└──────────────────────────────────────────────────────┘
Score = nesting_depth + cross_ref_count + function_count (each function/operator = 1)
Effective migration coverage:
Syntax-level: {N}/{total} ({%}%) — based on tier classification only
After orphan exclusion: {N}/{total} ({%}%) — minus {N} orphan inherited calcs
After dep resolution: {N}/{total} ({%}%) — minus {N} circular, {N} unresolvable cross-refs
──────────────────────
Realistic estimate: {N}/{total} ({%}%)
⚠ The tier classification reports SYNTAX-LEVEL translatability. The realistic
estimate accounts for orphan calcs (Step 3g), circular dependencies, and
unresolvable cross-references. A formula classified "Native" may still not
migrate if it depends on an orphan or circular chain.
Tableau Sets (top-level <group> elements — separate from calculated fields):
┌──────────────────────────────────────────────────────┐
│ Set tier Count Notes │
├──────────────────────────────────────────────────────┤
│ Native / column set {N} static + member-intersect → GROUP_BASED cohort (2a/2c) │
│ Query set {N} Top-N, condition, all-except-Top-N, mixed ops → ADVANCED (2b/2c) │
│ Partial / deferred {N} set controls + set actions (no equivalent) │
└──────────────────────────────────────────────────────┘
Parameters: {N} total ({N} static, {N} SQL-lookup — query at migration)
Dashboards: {N} (optional liveboard migration)
──────────────────────────────────────────────────
Migration coverage: {(all except untranslatable-ambiguous) / total}%
(all parameters auto-created — static or queried)
Untranslatable: {N} formula(s) — will be omitted
Geospatial: {N} formula(s) — spatial funcs omitted; lat/lon cols migrated as attributes
Deferred sets: {N} (set controls/actions — flagged for manual creation)
SQL-lookup params: {N} — need warehouse connection at migration time
Pass-through formulas (DENSE_RANK, SIZE, etc.) require SQL Passthrough Functions enabled.
Row-offset native formulas (LAG, LEAD, LOOKUP(agg,FIRST/LAST), INDEX) use native TS functions — no pass-through needed.
Bare FIRST()/LAST() standalone → omitted (returns offset, not value — no TS equivalent).
Migration effort estimate:
Formula translation: deterministic (CLI pipeline, no LLM calls)
Phase 1 import: ~1-2 minutes (tables + base model)
Phase 2 import: ~{N} retry cycles expected
Simple formulas: {N} — likely import on first try
Medium formulas: {N} — may need 1-2 retries
Complex formulas: {N} — expect structural fixes
Estimated total: {M}-{N} minutes (model only, excludes liveboard)
Orphan inherited calcs: {N} formula(s) referencing missing tables
┌──────────────────────────────────────────────────────────────────────────┐
│ These calcs were inherited from a parent/copied datasource and │
│ reference tables not present in this datasource. They are │
│ non-functional in Tableau and will be excluded from migration. │
│ │
│ Missing tables: {table1}, {table2}, … │
│ │
│ # Formula Name Missing Table Reference │
│ 1 {name} {TABLE_NAME} │
│ … │
│ │
│ (Includes {N} transitive orphans — depend on a direct orphan) │
└──────────────────────────────────────────────────────────────────────────┘
⚠ Formulas Needing Review: {N} formula(s)
┌──────────────────────────────────────────────────────────────────────────┐
│ # Formula Name Category What to Verify │
├──────────────────────────────────────────────────────────────────────────┤
│ 1 {name} No-keyword LOD Test with/without │
│ search filters │
│ 2 {name} Blend-context Compare totals │
│ 3 {name} Pass-through SQL SQL Passthrough on? │
│ 4 {name} ifnull stripped Verify null display │
│ 5 {name} sum_if rewrite Verify aggregation │
│ … │
├──────────────────────────────────────────────────────────────────────────┤
│ Categories: │
│ No-keyword LOD {AGG([col])} → group_aggregate — filter context may │
│ differ (Tableau: after dim filters, before table-calc)│
│ Blend-context Secondary datasource ref — post-agg blend vs row join│
│ Pass-through SQL sql_*_aggregate_op — requires cluster setting enabled│
│ ifnull stripped ifnull(measure,0) removed — TS handles NULLs natively│
│ sum_if rewrite sum(if…then…else 0) → sum_if() — semantically equiv │
└──────────────────────────────────────────────────────────────────────────┘
Data Blending — post-aggregation semantics:
┌──────────────────────────────────────────────────────────────────────────┐
│ ⚠ Tableau blends aggregate the secondary datasource independently │
│ before joining. ThoughtSpot model joins operate at row level. │
│ If both sides are fact tables at different grains, aggregation │
│ results may differ. │
├──────────────────────────────────────────────────────────────────────────┤
│ Primary DS Secondary DS Link Cols Risk │
├──────────────────────────────────────────────────────────────────────────┤
│ {ds_caption} {ds_caption} {col1, …} {HIGH/MED/LOW} │
│ … │
├──────────────────────────────────────────────────────────────────────────┤
│ Risk: HIGH = both sides have measures (fact×fact — aggregation may │
│ diverge) │
│ MED = secondary has measures but likely reference table │
│ LOW = secondary is dimension-only │
└──────────────────────────────────────────────────────────────────────────┘
Blended datasources: {N} of {total} — will merge into single model(s)
Blend relationships: {M} total
Star topologies: {S} (1 primary → 2+ secondaries)
HIGH-risk blends: {N} — verify aggregation results post-migration
──────────────────────────────────────────────────
Migration coverage includes everything except Untranslatable. All parameter types
are auto-migratable: static params are created directly in the model TML; SQL-lookup
params are populated by querying the warehouse at migration time. The formula reference
[Parameters].[Name] is rewritten to [Name] in both cases. Cite
references/coverage-matrix.md as the canonical source
when classifying a construct as mapped/unmapped or explaining a limitation.
If any formulas are classified as Row-offset, list them:
Row-offset formulas (translated via decision tree):
- {formula_name}: {tier} — {tableau_expr} → {ts_expr or "omit (ambiguous)"}
- ...
Excluded formulas — every formula that will NOT be migrated, grouped by root cause:
Excluded Formulas: {N} total
┌──────────────────────────────────────────────────────────────────────────┐
│ Root Cause Count Potential Resolution │
├──────────────────────────────────────────────────────────────────────────┤
│ Untranslatable function {N} No TS equivalent — SQL view or UDF │
│ Missing table in model {N} Add source tables, then create formula │
│ Circular dependency {N} Break cycle by inlining one formula │
│ Complex date arithmetic {N} Rewrite with TS date funcs or warehouse │
│ Geospatial function {N} Spatial not supported — lat/lon as attr │
└──────────────────────────────────────────────────────────────────────────┘
Per-formula detail:
{Root cause category} ({N}):
- {formula_name}: {tableau_expr} — {what the user can do}
- ...
If any SQL-lookup parameters exist, note them:
SQL-lookup parameters ({count} — populated from warehouse at migration time):
- {param_name}: query/column reference from TWB
- ...
Values are a point-in-time snapshot. Consider /ts-recipe-parameter-sync for
ongoing refresh.
If any formulas are classified as Pass-through, list them with the generated expression:
Pass-through formulas (require SQL Passthrough Functions enabled):
- {formula_name}: sql_{type}_aggregate_op("...", ...)
- ...
Write the report to /tmp/ts_tableau_mig/audit/{workbook_name}_audit.md and display
it inline.
Per-datasource breakdown (when a workbook has multiple datasources):
Report coverage per datasource, not just workbook-wide. Different datasources within
the same workbook can have very different effective migration rates (e.g. a full datasource
at 73% vs a copied subset at 54%). A combined number hides this.
Per-datasource coverage:
┌──────────────────────────────────────────────────────────────────────────┐
│ Datasource Tables Calcs Orphans Realistic Coverage │
├──────────────────────────────────────────────────────────────────────────┤
│ {ds_name_1} {N} {N} {N} {N}/{N} ({%}%) │
│ {ds_name_2} (copy) {N} {N} {N} {N}/{N} ({%}%) │
└──────────────────────────────────────────────────────────────────────────┘
Combined summary (multiple files):
Audit Summary: {N} workbook(s)
══════════════════════════════════════════════════════
Workbook Tables Calcs Orphans Coverage
─────────────────────────────────────────────────────────────────────
{workbook_1} {N} {N} {N} {%}%
{workbook_2} {N} {N} {N} {%}%
...
─────────────────────────────────────────────────────────────────────
Total {N} {N} {N} {%}%
Coverage = realistic estimate (after orphan + circular + cross-ref exclusion)
After the audit, exit cleanly. Do NOT proceed to Migrate mode steps.
Step 1 — Authenticate
Run ts profiles list to discover available ThoughtSpot profiles. If no profiles
exist, run /ts-profile-thoughtspot to create one. If multiple profiles exist,
display a numbered menu and ask the user to choose. If only one profile, use it
automatically.
ts profiles list
Then verify the chosen profile:
source ~/.zshenv && ts auth whoami --profile "{profile_name}"
Save {base_url} and {profile_name} for all subsequent steps.
Step 1.5 — Choose Migration Scope
Right after auth, ask what to migrate (this can be batched with the profile choice when
there are multiple profiles). The answer gates which later steps run:
What should I migrate?
1 Models + Liveboards — build/reuse tables & models, then dashboards → liveboards (default)
2 Tables + Models only — build the data layer; skip dashboards/liveboards
3 Liveboards only — models already exist; skip table/model creation and build
liveboards on an existing model I help you pick
4 Models only — tables already exist in ThoughtSpot; skip table creation,
build model(s) referencing existing tables
5 Tables only — generate and import table TMLs; skip models and liveboards
Migration pace? (scopes 1, 2, 4 only — omit for scopes 3 and 5)
F Fast — import formulas, park any failures, move on (default)
C Complete — after import, attempt to fix each failure (slower)
Enter scope (1-5) and pace (F/C):
Apply the scope:
| Scope | Runs | Skips |
|---|
| 1 Models + Liveboards | all steps | — |
| 2 Tables + Models only | 2–7.5, then 11.5 (coverage), 12 | 8–11 (dashboards/liveboards) |
| 3 Liveboards only | 1.5a model picker, 2–3 (parse, dashboards), 8–12 | 4–7.5 (table/model creation) |
| 4 Models only | 2–3 (parse), 4 (E/G to find existing tables), 5b (model TML + formulas), 5.5, 6, 7, 7.5, 11.5, 12 | 4.5 (connection — tables already bound), 5a (table TMLs), 8–11 (liveboards) |
| 5 Tables only | 2–3 (parse), 4, 4.5, 5a (table TMLs), 6, 12 | 5b (model), 5.5–7.5, 8–11 |
Save the pace as {migration_pace} (F or C). Default F if the user enters only a
scope number or skips the pace question. For scopes 3 and 5, {migration_pace} is always
F (no formula imports, so the pace has no effect).
For scope 2, after Step 7.5 jump straight to Step 11.5 then Step 12.
For scope 3, there is no model to build — the user selects an existing model, and the
liveboard tiles reference its columns/formulas. Run Step 1.5a below to pick it, then
parse the TWB (Steps 2–3) and continue at Step 8. (Step 9b maps each worksheet's shelves to
the chosen model's columns by display name — surface any field that has no matching
column rather than guessing.)
For scope 4, tables already exist in ThoughtSpot — the user provides GUIDs or searches
for them (Step 4, E or G path). No connection selection or table TML generation
needed. The model TML references existing tables by GUID. This is the common path for
consultant/remote migrations where tables were loaded separately (e.g. via
/ts-load-source-data or manual warehouse provisioning + ts tables create). After
Step 7.5, jump to Step 11.5 then Step 12.
For scope 5, only table TMLs are generated and imported. No model, no formulas, no
liveboard. Useful for a phased migration where tables are set up first, then models are
created in a second pass (scope 4).
Step 1.5a — Pick an existing model (scope 3 only)
A ThoughtSpot Model is a worksheetVersion: V2 logical table — there is no MODEL
subtype in metadata search. Find models with --subtype WORKSHEET (which returns
worksheets and models) and keep only those whose metadata_header.worksheetVersion == "V2".
Prompt how to identify the model — don't list every model by default (the full list is
slow on a large instance). Mirror the connection picker (Step 4.5):
How would you like to choose the model?
G GUID — paste the model's GUID; I'll fetch it directly (fastest)
N Name it — type the exact model name
F Filter — give a partial string; I'll list matching models
L List all — show every model and pick by number (slow — scans all worksheets)
Enter G / N / F / L:
Resolve the choice:
source ~/.zshenv && ts metadata search --subtype WORKSHEET --name "%{partial}%" --profile {profile_name}
From the chosen model capture and save: {model_guid} (metadata_id), {model_obj_id}
(metadata_obj_id — the real obj_id; see the obj_id rule in Step 7), and {model_name}
(metadata_name). Then export it once (Step 10-pre) to read its columns, formulas, and
parameters[].id for building liveboard tiles. Confirm the picked model with the user before
proceeding:
Using existing model: {model_name}
{base_url}/#/data/tables/{model_guid}
Columns: {n} Formulas: {f} Parameters: {names}
Build liveboards on this model? (yes / pick another)
Step 2 — Extract TWB File
Ask for the file path if not yet provided.
If the file ends in .twbx (a ZIP archive), extract it:
mkdir -p /tmp/ts_tableau_mig && unzip -o "{twbx_path}" -d /tmp/ts_tableau_mig/
Then find the .twb inside:
find /tmp/ts_tableau_mig -name "*.twb" | head -1
Save the resolved .twb path as {twb_path}.
Step 3 — Parse TWB XML
Run the parser and read its JSON — do NOT hand-parse the XML:
ts tableau parse "{twb_path}" --output /tmp/ts_tableau_mig/{workbook_name}_parsed.json
The JSON contains datasources[] (each with tables, columns, joins,
calculated_fields, calc_map, col_table_map, orphan_calcs), parameters,
param_map, blends, and table_calc_addressing. All subsequent steps read
these fields instead of re-deriving them.
The parse output (from the ts tableau parse call above) contains the following, extracted from the TWB's XML structure:
3a. Workbook name
Take from the filename (strip .twb). Save as {workbook_name}.
3b. For each <datasource> element (skip those named Parameters)
Each datasource is processed independently for extraction (Steps 3b–3d). Datasources are
merged into a single model only when they are connected by blend relationships (detected in
Step 3e). Even when datasources share tables or point at the same database, they are NOT
merged unless a <datasource-relationship> explicitly links them. See Step 5b
"Blend-aware model grouping" for the merge procedure.
Datasource type detection:
- If the datasource contains
<connection class="sqlproxy">, it is a Published
Datasource (hosted on Tableau Server). The table name resolves to
connection.get('dbname'), not the literal [sqlproxy].
- If the datasource contains
<extract>, do not blindly skip it. An extract is a
local snapshot, but it almost always wraps an underlying connection that names a real
table — a file source (textscan/CSV, excel-direct), a database, etc. What matters
for migration is that underlying source, because that's what gets queried in the
warehouse. Look past the <extract>/hyper connection to the real one:
- The relation has two parents in
<metadata-records> — the live source (e.g.
[Amazon Sales data.csv]) and [Extract]. Use the live-source relation; ignore the
[Extract] relation. The physical table name comes from the live source (mapped to
its warehouse table per Step 4.5).
- Only treat a datasource as truly skippable when there is no resolvable underlying
connection (a pure Tableau-authored extract with no source) — and say so in the report.
- File-based sources (CSV/Excel) imply the data was loaded into the warehouse out of
band; bind the table to the connection that now exposes it (Step 4/4.5).
- Otherwise, it is a Live datasource — proceed with extraction.
Non-warehouse sources — explicit unsupported policy: The following Tableau connection
classes are NOT warehouse-bound and cannot be mapped to a ThoughtSpot connection:
cloudfile:googledrive-excel-direct, google-sheets, ogrdirect (spatial/OGR),
webdata-direct (web data connector), CustomMapbox. When any of these appear as a
datasource's connection class, do NOT assume a warehouse table exists. Instead:
- Log:
"Datasource '<name>' uses a non-warehouse source (<class>) — cannot map to a ThoughtSpot connection. Skipped; data must be loaded into a warehouse first."
- Skip the datasource entirely (do not generate table or model TML for it).
- Surface in the audit report under a "Skipped sources" section.
Redshift and Postgres dialect notes: When <connection class="redshift"> or
<connection class="postgres"> is detected, pass-through SQL (sql_*_op) formulas
should use the corresponding dialect syntax. Key differences from Snowflake:
- String concatenation:
|| (same as Snowflake)
- Date truncation:
date_trunc('month', col) (same syntax, both dialects)
LISTAGG → Redshift: LISTAGG(col, ',') WITHIN GROUP (ORDER BY col); Postgres: string_agg(col, ',' ORDER BY col)- Type casting: Redshift uses
::type; Postgres uses CAST(x AS type) or ::type
No other mapping changes are needed — the Tableau-to-ThoughtSpot formula translation is
warehouse-agnostic (ThoughtSpot formulas are the target, not SQL). The dialect only matters
for sql_*_op pass-through functions.
Relation wrapper handling: TWB XML wraps <relation> elements in one of three
structures. Check in order:
_.fcp.ObjectModelEncapsulateLegacy.false...relation tag_.fcp.ObjectModelEncapsulateLegacy.true...relation tag<relation> directly under <connection class='federated'> (fallback)
All three contain the same child elements — the wrapper determines where to look.
For each datasource, extract:
Physical tables — <relation> elements of type="table":
name attribute = table alias used in joinstable attribute = fully-qualified physical table name — may be [DB].[SCHEMA].[TABLE]
format; strip brackets and split on . to extract db, schema, and table components- For Published Datasources (sqlproxy): if table name is
[sqlproxy], use
connection.get('dbname') instead
Custom SQL relations — <relation> elements of type="text":
- These contain raw SQL in the element text content — do NOT try to extract a table name
- Flag the relation as
source_type: "custom-sql" and save the full SQL text
- Refactor the SQL: replace
<< with <, >> with >, == with = (XML encoding
artifacts from the TWB)
- These will generate a
sql_view: TML instead of a table: TML (see Step 5c)
- Extract column names from the SQL
SELECT clause aliases for column mapping
Joins — <relation> elements of type="join":
join attribute = join type (inner | left | right | full)<clause> child = join condition (decode HTML entities: "→",
&→&, <→<, >→>)- Extract left and right table references from the clause
Physical columns — from <metadata-records> → <metadata-record class="column">:
local-name = column identifierremote-name = physical column name in the database (use for db_column_name)local-type = Tableau data typeparent-name = which table this column belongs to- Also extract from
<column> elements WITHOUT a <calculation> child:
name (strip brackets), datatype, role (dimension/measure), caption (display name)
Calculated fields — <column> elements WITH a <calculation class="tableau"> child:
- Skip columns where
param-domain-type is list or range — these are Tableau
parameters, not calculated fields
caption or name = display namecalculation formula attribute = Tableau expression (decode HTML entities)datatype attribute- Build a cross-reference map: Tableau internal names (
[Calculation_1234567890]) →
display names. Calculated fields reference each other by internal ID in the TWB XML,
not by display name — resolve these references before translating formulas.
Parameters — <datasource name="Parameters"> children:
- For each
<column> with param-domain-type attribute:
caption = display name (used as ThoughtSpot parameter name)datatype = string | integer | real | date | booleanparam-domain-type = list | range | anyvalue attribute or calculation.formula = default value<member value="..."> children = list values (when param-domain-type="list")<range min="..." max="..."> child = range bounds (when param-domain-type="range")
- Save parameter definitions — these generate
model.parameters[] in Step 5b
- SQL-lookup parameters (where the list values come from a database query rather
than static
<member> elements): save the query/column reference — at migration
time (Step 5b), query the warehouse to populate list_config.list_choice[] with
current values. In audit mode (no connection), flag as "requires connection"
Save the parsed structure internally. Announce a summary:
Parsed {workbook_name}: {N} datasource(s), {N} physical table(s),
{N} calculated field(s), {N} join(s), {N} dashboard(s)
3c. Topological sort of calculated fields
Some calculated fields reference other calculated fields. Sort them so that fields
with no formula-dependencies come first (Level 0), then Level 1, etc. This determines
the order they must appear in the model TML formulas section.
Resolve all internal Tableau cross-references ([Calculation_\d+] → display name)
before sorting. The topological sort must use display names, not internal IDs.
3d. Dashboard metadata (for Step 8 decision)
Count <dashboard> elements in the TWB. Save the count and names — this is shown
in Step 8 when asking whether to migrate dashboards.
3e. Extract blend relationships (data blending)
Parse the <datasource-relationships> element at the workbook root (child of <workbook>).
If absent, no blending is used — skip this step.
Build the blend graph:
The parse-JSON blends field (Step 3) is this graph — {source_ds_caption: [{target_ds, column_mappings}]}, with federated IDs already resolved to
datasource captions (matching each datasource's name from Step 3b). Treat
blend_graph in the rest of this document as shorthand for that field.
What to log:
- Number of blend relationships found
- For each: primary datasource → secondary datasource, with linking columns listed
Date-grain linking columns:
When a <column-instance> has a derivation other than "None" (e.g. "Month",
"Month-Trunc", "Year", "Year-Trunc"), the blend links at a specific time grain.
For the ThoughtSpot model join, the physical date column is used directly — ThoughtSpot's
date bucketing at query time handles the grain alignment.
However, if the source and target columns are physically different date columns with
different native grains (e.g. source has daily Order Date, target has monthly
Month of Order Date that is already pre-truncated), the join requires a
date-truncation formula or SQL View to materialize the matching grain.
Resolution strategy:
- If both columns are date/datetime type and the derivation indicates a truncation
(
Month-Trunc, Year-Trunc), emit a model formula:
date_trunc ( 'month' , [TABLE::Order Date] ) and use that formula as the join key
via a SQL View (the formula can't be a direct join key in model TML).
- Surface the grain mismatch to the user in the review checkpoint with a recommendation:
- "Blend links
Order Date (daily) to Month of Order Date (monthly) at month grain.
Recommend: create a SQL View with DATE_TRUNC('MONTH', ORDER_DATE) AS ORDER_MONTH and
join on ORDER_MONTH = MONTH_OF_ORDER_DATE."
- If both columns are the same physical type and grain, use them directly in the join
on
clause — no materialization needed.
No model merging happens here — this step only extracts the relationships. Model
merging happens in Step 5b.
Step 3f — Table-calc addressing extraction
For every datasource, scan <column> elements that have a <calculation class='tableau'>
child containing a <table-calc> element. This is the column-level addressing map —
read it from the parse-JSON table_calc_addressing.column_level field (Step 3): keyed by
calc internal ID (e.g. [Calculation_953355781789577216]), each entry has
ordering_type (Rows | Columns | Table | CellInPane | Field), ordering_field,
order_fields (list), quick_calc_type (PctTotal | PctDiff | Difference |
PctRank | None), and address_offset (int or None).
Each <worksheet>'s <column-instance> elements can carry their own <table-calc> —
these are view-level overrides that take precedence over the column-level definition
for that worksheet. Read them from table_calc_addressing.ws_overrides (same entry shape,
keyed by worksheet name then calc ID).
Resolution order when translating a table-calc formula used on worksheet W:
- Check
table_calc_addressing.ws_overrides[W][calc_id] — view-level override
- Fall back to
table_calc_addressing.column_level[calc_id] — column-level definition
- If neither exists, treat as
ordering_type='Rows' (Tableau default)
Step 3g — Orphan calc detection (copied datasources)
When a Tableau datasource is a copy of another (common with published datasource
clones), it inherits all calculated fields from the original — including ones that
reference tables no longer present in the copy. These orphan calcs are non-functional in
Tableau and will fail at ThoughtSpot import.
Each datasource's orphan_calcs (parse-JSON field, Step 3) is this list — captions of
calcs that directly reference a table missing from the datasource, plus calcs that
transitively depend on a direct orphan.
What to log (if any orphans found):
⚠ Datasource {name}: {N} orphan calc(s) reference missing tables: {table1, table2, …}
These are non-functional inherited fields and will be excluded from migration.
In migrate mode: surface the orphan count and the missing tables before proceeding.
Ask the user to confirm exclusion — in rare cases they may want to add the missing tables
to the connection and model instead:
{N} calculated fields reference tables not in this datasource ({table1, table2, …}).
These appear to be inherited from a parent datasource and are non-functional.
E — Exclude them (default, recommended)
A — Add the missing tables to the model (you'll need to confirm they exist in Step 4)
If the user chooses A, add the missing tables to the datasource's table list and
re-run Step 4 table confirmation for the new tables only. Remove the affected calcs
from orphan_calcs so they enter the translation pipeline.
In audit mode: no prompt — just report the orphan count in the A4 audit report.
Step 3.5 — Resolve Published Datasources (sqlproxy)
Runs only if Step 3 detected one or more datasources with <connection class="sqlproxy">.
Skipped entirely if all datasources have direct warehouse connections.
When a Tableau workbook references a published datasource on Tableau Server/Cloud,
the TWB XML contains <connection class="sqlproxy"> with a dbname attribute naming the
published datasource.
What the TWB DOES contain for sqlproxy datasources (extract these in Step 3b regardless
of API access — they are <column> elements directly under the <datasource> element):
- All calculated fields with full Tableau formula text (same as direct-connection
datasources)
- Column definitions with captions, datatypes, and roles
- Metadata records (local-name, remote-name, local-type, parent) — often complete enough
for column mapping
What the TWB does NOT contain (it lives only in the published datasource's .tds):
- The physical table structure (table names, joins, db/schema/table paths)
- The connection details (database, schema) that link to the warehouse
This means formula extraction and translation work without the physical model — the
.tds adds physical table resolution and join definitions, not the formulas.
Where the physical model actually is — and how to get it. The join/table structure is
not returned by the field API. ts tableau datasource --fields (VizQL read-metadata)
returns columns/calcs only, not tables or joins. The full physical model (tables, joins,
custom SQL) lives in the published datasource's .tds. Two ways to obtain and parse it:
- Download it (needs Tableau access):
ts tableau download {datasource_id} fetches the
.tdsx; the .tds inside carries the model.
- Be supplied it: the user provides the
.tds/.tdsx alongside the .twb.
Then ts tableau parse accepts a .tds/.tdsx directly (ts-cli ≥ 0.38.0) — its root
is the <datasource>, and parse extracts its tables/joins/columns/calcs just like a
workbook datasource. Feed that to build-model GENERATE mode and it builds the multi-table
model automatically — no hand-assembly (see Step 5b "Multi-query datasources"). Without
the .tds (only the .twb, no Tableau access), fall back to the hand-built multi-table base
in Step 5b.
Flow
ASK before querying the Tableau API. The user may be a consultant conducting a remote
migration without access to the customer's Tableau Server. Do NOT attempt any API call
before asking — a failed API call wastes 30–60 seconds and confuses the flow.
-
Prompt — always, before any API call:
Found {N} published datasource(s) hosted on Tableau Server:
- {ds_caption_1} ({M} columns, {C} calculated fields extracted from TWB)
- {ds_caption_2} ({M} columns, {C} calculated fields extracted from TWB)
The TWB already contains all column definitions and calculated fields.
The Tableau API would additionally resolve the physical table structure
(table names, joins, db/schema paths) — but this is optional.
Do you have access to the Tableau Server hosting these datasources?
Y Yes — query the Tableau API for table structure (requires /ts-profile-tableau)
N No — proceed with TWB metadata only (common for consultant/remote migrations)
Enter Y / N:
-
If the user chooses N (no API access):
- Log:
"Proceeding with TWB-embedded metadata — {M} columns, {C} calculated fields already extracted. Physical table names and joins will be confirmed manually in Steps 3.6 and 4."
- The TWB's
<column> elements (captions, datatypes, roles) and <metadata-records>
(local-name → remote-name mapping) provide enough for formula translation and column
mapping. The physical table names come from <metadata-record> parent-name attributes
(e.g. [Custom SQL Query3], [Table_Name]).
- Skip to Step 3.6 (join confirmation) — the user will provide or confirm joins and
table mappings manually. This is the normal path for consultant/remote migrations.
- Continue to Step 4 with the TWB column info.
-
For each sqlproxy datasource, extract dbname from the <connection> element, then:
Progress label: "Querying Tableau API (not ThoughtSpot) to resolve published datasource columns…"
— make it clear this is a Tableau Server query, not a ThoughtSpot metadata search.
ts tableau datasources --profile {PROFILE} --name "{dbname}"
Parse the JSON output to get the datasource id.
ts tableau datasource {id} --profile {PROFILE} --fields
The fields array contains:
| Field | Use |
|---|
fieldCaption | Column display name → ThoughtSpot column name |
dataType | real/integer/string/date/datetime/boolean → TS data type |
columnClass | COLUMN (physical), CALCULATION (formula), BIN, GROUP |
formula | For calculated fields — the Tableau formula text for Step 5 translation |
-
Merge the resolved fields into the parsed datasource structure, replacing opaque
sqlproxy column references with real names and types. Proceed to Step 4.
-
For textscan (CSV) or excel-direct sources: offer to download the source data
for warehouse provisioning. This is essential when the data only exists in Tableau Cloud
and has not been loaded into a warehouse:
ts tableau download {datasource_id} --profile {PROFILE} --output-dir {output_dir}
The command downloads the TDSX archive, extracts it, and validates CSV files for row
integrity (column count consistency, corrupt lines). Check the validation result:
- If
is_valid: false — report the corrupt lines and offer to auto-fix (strip them)
before proceeding to warehouse load. The DunderMifflin live test (2026-06-26) found a
corrupt line (1tou) in a Tableau Cloud textscan download — this is a known Tableau
artifact.
- If
is_valid: true — proceed; the CSV is clean for loading.
Surface: "The data for datasource '{name}' is a {type} file hosted on Tableau Cloud. It
needs to be loaded into a warehouse table before ThoughtSpot can connect to it. I've
downloaded and validated it — {row_count} rows, {status}. Shall I help set up the warehouse
table? (This will require a Snowflake/Databricks connection.)"
If the user says yes, this is the handoff point for BL-010 (ts-load-source-data) when
that skill is built. Until then, guide the user through manual warehouse provisioning
(CREATE TABLE + stage + COPY INTO for Snowflake, or INSERT VALUES for Databricks).
Prerequisites
- Tableau profile configured via
/ts-profile-tableau (optional — skill degrades gracefully)
ts CLI v0.29.0+ (includes ts tableau build-model with --max-retries, enriched error
reporting, GENERATE mode phased output, and --table-name-map)
Step 3.6 — Join Confirmation
Joins define the model's query behavior — never silently add joins that aren't in the TWB.
When the TWB parse found joins (from <relation type="join"> or <object-graph><relationships>)
Present them to the user for confirmation:
Joins found in workbook ({N} total):
1. TABLE_A LEFT JOIN TABLE_B ON TABLE_A.COL = TABLE_B.COL
2. TABLE_A LEFT JOIN TABLE_C ON TABLE_A.COL = TABLE_C.COL
Do these look correct? (Y/N/Edit)
If the user edits, accept updated join definitions and continue.
When the TWB parse found NO joins (common with published datasources/sqlproxy)
⚠ No join definitions found in the workbook file.
This is normal for published datasources — joins are defined server-side.
Tables in this datasource: TABLE_A, TABLE_B, TABLE_C, ...
To build the model, I need join definitions. Options:
D Define joins — I'll suggest based on matching column names, you confirm
S Skip joins — create separate single-table models (no multi-table queries)
P Provide — paste or describe join definitions
If the user picks D, suggest joins based on shared column names between tables:
Suggested joins (based on shared column names):
TABLE_A.PROMOTION_ID = TABLE_B.PROMOTION_ID (LEFT_OUTER, MANY_TO_ONE)
TABLE_A.PROMOTION_ID = TABLE_C.PROMOTION_ID (LEFT_OUTER, MANY_TO_ONE)
Accept suggested joins? (Y/N/Edit)
Range join alternative
When the parse detects date-range filter formulas (e.g., [DATE] >= [START_DATE] AND [DATE] <= [END_DATE]) AND separate fact/dimension tables with start/end date columns,
surface an additional option:
💡 Detected date-range filter pattern. ThoughtSpot supports range joins:
FACT.DATE >= DIM.START_DATE and FACT.DATE < DIM.END_DATE
This is more efficient than a filter formula. Use a range join instead? (Y/N)
See tableau-tml-rules.md "Range join alternative" for the TML syntax.
Step 4 — Confirm Source Tables (ask before searching)
This is the first thing after the parse — before selecting a connection, searching
ThoughtSpot, or fetching any schema. Getting the order wrong wastes the user's time:
scanning the whole instance, or pulling a connection's schema, when the user already knows
whether the tables exist is pure overhead (and the connection-schema fetch is slow and can
404). Ask first; act second. This step only asks and confirms — it never loads or
modifies warehouse data. It mirrors ts-convert-from-databricks-mv Step 7.
Do NOT run ts metadata search, ts connections list, or ts connections get until
the user has answered the question in 4a. No exploratory "let me just check" searches —
the answer decides whether any search runs, and at what scope. An ungated
ts metadata search --all on a large instance is exactly the wasted work this step exists
to prevent.
4a — Present the table inventory and ask
Scope 4 (Models only): tables already exist by definition. Skip choices N and ? —
present only E and G and default to G if the user has the GUIDs. The user chose
scope 4 specifically because the tables are in ThoughtSpot already.
Show the full inventory of physical tables from Step 3, then ask whether they already exist
as ThoughtSpot Table objects:
Source tables referenced by {workbook_name} ({N} total):
1. P1-UK-Bank-Customers
…
Do these already exist as ThoughtSpot Table objects?
E Exist — reuse them (I'll search for their GUIDs)
G Have GUIDs — provide GUIDs directly (fastest, no search needed)
N Don't exist — create new on a connection (default)
? Not sure — search ThoughtSpot to check (avoids creating duplicates)
Enter E / G / N / ? :
If the tables differ in status (some exist, some don't), accept a per-table answer.
4b — Act on the answer
-
N (don't exist) → no search. Go to Step 4.5 to pick the connection, then
create Table TMLs in Step 5a (the default path).
Deduplication note: if you are migrating multiple workbooks that share the same
published datasource, the tables from the first migration already exist. Choosing N
again creates duplicates. If this is a second (or later) workbook migration, consider
E, G, or ? instead to reuse the tables already in ThoughtSpot.
-
G (have GUIDs) → no search. For each table, ask the user to provide the GUID
(the id value from ThoughtSpot, e.g. from a previous migration or from the UI). Use
the provided GUIDs in the model's model_tables[] entries and skip generating Table
TMLs for those tables. If the user has GUIDs for some tables but not all, treat the
remaining tables as N (create via Step 4.5 + 5a).
-
E (exist) → search to find the GUIDs — but choose the scope first (4c).
Searching ensures the tables are not duplicated and resolves their GUIDs automatically.
-
? (not sure) → search — choose the scope first (4c). Report what was / wasn't
found; reuse the found ones, treat not-found tables as create (Step 4.5 + 5a).
This is the safest option when migrating into an instance that may already have
some of these tables.
4c — Choose the search scope (E and ? paths only)
A whole-instance scan is the slow path. Always offer the narrower option, and search by
table-name pattern (--name) so the API does the filtering — never pull every table
and filter locally:
How should I search for these tables?
C Within a specific connection — fastest; I'll list connections and search that one
I Entire ThoughtSpot instance — broader, slower
Enter C / I :
source ~/.zshenv && ts metadata search --subtype ONE_TO_ONE_LOGICAL --name "%{table_name}%" --profile {profile_name}
- C (within a connection) → first identify the connection using the
N (name it) / F (filter by substring) / L (list all) prompt in Step 4.5 — present that
prompt and let the user choose; do NOT run
ts connections list and dump every connection
by default. Once the connection is chosen, run the name search above and keep only
results whose metadata_header.dataSourceName equals the chosen connection name
(verified 2026-06-16: each search result carries its connection in
metadata_header.dataSourceName, e.g. "APJ_TAB"). Fastest, and unambiguous when the
same table name exists on several connections.
- I (entire instance) → run the name search above with no connection filter.
Match on table name (metadata_name) and, for the connection scope,
metadata_header.dataSourceName. (db/schema also appear in metadata_header —
database_stripes / schema_stripes — use them to disambiguate same-named tables within
one connection.) For each table found, reuse its
name/GUID in the model's model_tables[] and skip generating a Table TML for it in
Step 5a. If a table the user said Exists is not found, say so and confirm before falling
back to create.
Only fall back to --all (fetch every table) when no usable name pattern can be formed
(e.g. the name is too generic). Tell the user that cost before running it.
4d — Confirm any missing sources before proceeding
If any table the plan intends to create is not found on the chosen connection,
surface it and require confirmation — this is the silent-failure case (a model TML that
points at a table the connection can't see still imports cleanly, yet every search and
liveboard built on it comes back empty):
⚠ The following table(s) are not visible to connection "{connection_name}":
- {db}.{schema}.{db_table}
Their models will import, but searches return no data until the data is loaded
and visible to the connection. This skill does not load data.
Proceed anyway (generate the TMLs as-is)? (yes / no):
Do not proceed past this warning without the user's confirmation.
Step 4.5 — Select ThoughtSpot Connection (create path or connection-scoped search)
Run this only when a table will be created (the N path, or tables not found on the
E / ? paths) or to scope a connection search in 4c. If every table was matched to
an existing object, skip this step — reusing tables needs no connection work.
First: use an existing connection or create a new one. Ask:
The generated tables need a ThoughtSpot connection that can reach the source database.
E Use an existing connection
C Create a new connection (Snowflake source only, key-pair auth)
Enter E / C:
When to create: a ThoughtSpot connection only sees databases its warehouse
role is granted. If no existing connection's role can see the source database,
table creation fails with "Database … does not exist in connection" — that is the
signal to create one (do not trial-and-error existing connections to find out).
C — create a new connection. Supported here for Snowflake sources via key-pair
auth. Collect the connection name, Snowflake account identifier, user, role, warehouse,
and the path to the unencrypted PKCS#8 private key (.p8), then run:
source ~/.zshenv && ts connections create \
--name "{connection_name}" \
--account "{account}" --user "{user}" --role "{role}" --warehouse "{warehouse}" \
--database "{database}" \
--private-key-path "{key_path}" \
--profile {profile_name}
The role must have USAGE on the database/schema (and SELECT on the tables). The
matching public key must already be registered on the Snowflake user. Credential
handling (required): never ask the user to paste a private key, password, or secret
into the conversation — the key is passed by file path only and ts connections create never echoes it. If the source is not Snowflake, or password/OAuth is
required, connection creation is out of this skill's scope: direct the user to create
the connection in the ThoughtSpot UI, then return on the E path. Use the returned
name as {connection_name}.
E — use an existing connection. Don't dump the full list by default — a long
connection list is noise when the user already knows the one they want. Ask:
How would you like to identify the connection?
N Name it — type the exact connection name; I'll use it directly
F Filter — give a partial string; I'll list only connections that match
L List all — show every connection and pick by number
T Trust — type the name and skip validation (faster — import will fail
cleanly if the name is wrong)
Enter N / F / L / T:
T — trust the name. Use the typed name directly without running ts connections list.
This skips validation but is faster on instances with many connections. The import will
return a clear error ("Connection 'X' not found") if the name is wrong.
Compound prompt (N or T path). When the user takes the N or T path, offer the
db/schema confirmation in the same prompt to eliminate sequential questions:
Connection: ____________ (exact ThoughtSpot connection name)
Database: ____________ (or press Enter to use '{twb_extracted_db}')
Schema: ____________ (or press Enter to use '{twb_extracted_schema}')
This replaces the separate db/schema confirmation loop below when the user provides
all three in one response.
For N/F/L, fetch the connections once (auto-paginated, returns all):
source ~/.zshenv && ts connections list --profile {profile_name}
Resolve the user's choice against that result:
-
N (name it) — match the typed name against the returned name values
(case-sensitive). Exactly one match → use it. No match → show the closest names and
re-ask. Don't fabricate a name the list doesn't contain — the table TML needs the exact,
case-sensitive connection name.
-
F (filter) — keep connections whose name contains the string (case-insensitive),
show them as a short numbered list (name, type, database), and pick from that. One match
→ auto-select and confirm; none → widen the string or switch to L.
-
L (list all) — show the full numbered list and pick by number:
Available ThoughtSpot connections:
1. SNOWFLAKE_PROD (RDBMS_SNOWFLAKE) — PROD_DB
2. ANALYTICS_DW (RDBMS_SNOWFLAKE) — ANALYTICS_DB
Which connection should the generated tables use? (Enter number):
If only one connection exists in total, auto-select it and confirm regardless of the choice.
Save the selected connection's exact name value as {connection_name}.
Resolving db / schema / table for new tables. Each new table needs the {db},
{schema}, and {db_table} it maps to on the chosen connection. The Tableau workbook
contains the source environment's database paths — these may not match the target
ThoughtSpot connection (e.g. a consultant running the migration in their own environment
with a different database). Always confirm before using them.
Show the TWB-extracted paths and ask:
The Tableau workbook references these source database paths:
- {source_db}.{source_schema}.{table_1}
- {source_db}.{source_schema}.{table_2}
…
Do these match your ThoughtSpot connection's database and schema?
Y Yes — use these paths as-is
D Different database/schema — I'll provide the correct values
T Per-table — some match, some don't (I'll confirm each)
Enter Y / D / T:
- Y → use the TWB-extracted
{db}, {schema}, and {db_table} values directly.
- D → ask for the target
{db} and {schema} once. Apply them to all tables (table
names stay the same unless the user overrides). This is the common consultant scenario
where all tables live in the same database but under a different name.
- T → walk through each table and confirm or override its
{db}, {schema}, and
{db_table}. Use this when tables span multiple databases or schemas in the target.
If the user doesn't know the correct paths:
- Ask the user for the db / schema (and table name if it differs from the source) —
usually instant, and they know it.
- Only if they're unsure, fetch the connection schema to resolve names:
source ~/.zshenv && ts connections get {connection_id} --profile {profile_name}
This uses the v1 fetchConnection endpoint — it can be slow and returns 404 on some
connection types, so treat it as the fallback, not the default. If it returns no
tables (empty externalDatabases) or fails, ask the user for the names directly.
A connection is required for any table being created — there is no skip path.
ThoughtSpot tables are logical objects over a live connection: the physical table must
already exist in the database and the connection must already exist for the table to be
created at all. Do not offer placeholders or a dry-run mode — they only produce objects
that can never bind to data. If the user has no suitable connection: for a Snowflake
source, create one via the C path above (key-pair auth); for any other source, or when
password/OAuth is required, stop and tell them the connection must be created first in the
ThoughtSpot UI (that connection setup is out of this skill's scope).
Use the connection's exact name in every table TML and SQL View TML — never a GUID. The
v2 API cannot search connections by name, so the name string is both necessary and
sufficient; do not try to resolve it to an ID. See
../../shared/schemas/thoughtspot-table-tml.md "Connection Reference".
Step 5 — Generate TML Files
Create output directory:
mkdir -p /tmp/ts_tableau_mig/output/{workbook_name}
5a. Table TML — one per physical table (skip custom SQL relations)
Scope gate: runs for scopes 1, 2, 5. Skip for scope 3 (LB only — no tables)
and scope 4 (Models only — tables already exist; use GUIDs from Step 4).
For each physical table identified in Step 3 with type="table", generate a
.table.tml file. Skip custom SQL relations — those are handled in Step 5c.
Follow all rules in tableau-tml-rules.md.
Template:
table:
name: TABLE_NAME
db: RESOLVED_DATABASE
schema: RESOLVED_SCHEMA
db_table: physical_table_name
connection:
name: "{connection_name}"
columns:
- name: COLUMN_NAME
db_column_name: COLUMN_NAME
data_type: VARCHAR
properties:
column_type: ATTRIBUTE
aggregation: SUM
db_column_properties:
data_type: VARCHAR
Key rules:
connection.name is required — a ThoughtSpot logical table must sit on a connection
that already exposes the physical table and its columns. Use the connection name
directly (case-sensitive); never look up a GUID — the v2 API cannot search connections
by name, and the name is what the TML needs. See ../../shared/schemas/thoughtspot-table-tml.md
"Connection Reference".- Use the
db, schema values resolved from Step 4.5 (the connection is required, so these
are always real).
db_column_name must match the physical column the connection exposes — not the
Tableau name. When a file source (CSV/Excel) was loaded into the warehouse, the loader
usually normalizes names (Item Type → ITEM_TYPE: spaces→_, upper-cased). Use the
warehouse column name for db_column_name (and the friendly Tableau caption for the
model column's display name). If unsure, the connection schema from Step 4.5
(externalDatabases) lists the real column names; validation reports
column not found in connection when they don't match.- Date stored as VARCHAR — flag it. If the Tableau column is typed
date/datetime
but the warehouse column is VARCHAR (common when a CSV date loaded as text), binding
it as VARCHAR loses all date capability (no buckets/trends/relative-date filters; Spotter
won't read it as time). The TS column data_type must match the physical column, so you
can't just declare DATE. Surface it and offer: (a) retype at the source (warehouse
ALTER/reload to a real DATE — outside this skill; needs the user) then bind as DATE,
or (b) keep VARCHAR and add a to_date([col]) derived formula column for
date analytics. Don't silently bind a date as a string.
- Partial date strings must produce a full
YYYY-MM-DD date. When a source column
contains a year-only value (e.g. _2016_17, FY2016, 2016) and needs to become a
DATE, always append -01-01 (or -01 for year-month) to produce a complete date.
A bare-year conversion like to_date('2016', 'yyyy') produces an ambiguous value that
ThoughtSpot cannot bucket (.yearly, .monthly), use for KPI sparklines, or filter as
a date range. If the datasource already uses a SQL View, apply the conversion in the
SQL query (TO_DATE(SUBSTRING(col, 2, 4) || '-01-01', 'YYYY-MM-DD')). If it uses a
regular table, apply it as a model formula
(to_date ( concat ( substr ( [col] , 1 , 4 ) , '-01-01' ) , 'yyyy-MM-dd' )). See tableau-tml-rules.md
"Date Column Rules" for the full pattern table.
- Use
INT64 for Tableau integer — never INT
