ts-convert-from-tableau

name	ts-convert-from-tableau
description	Convert or import a Tableau workbook (.twb or .twbx) into ThoughtSpot — parses TWB XML, generates table + model TMLs, validates and imports. Optionally migrates dashboards to liveboards with layout approximation. Direction is always Tableau → ThoughtSpot. Not for ThoughtSpot → Tableau or standalone TML exports.

Tableau Workbook → ThoughtSpot

Converts a Tableau workbook into ThoughtSpot objects. Parses the TWB XML to extract tables, columns, joins, and calculated fields, then generates Table TMLs and a Model TML per datasource. Optionally converts Tableau dashboards into ThoughtSpot Liveboards with approximate layout mapping.

Ask one question at a time for dependent decisions (where the next question depends on the answer), waiting for each. But batch independent questions into a single multi-question prompt to cut round-trips and keep the migration fast — e.g. mode + scope, the count-column + bin-style + cohort-handling decisions, or theme + parameter-chips. See Efficiency in Step 0.

---

References

File	Purpose
../../shared/mappings/tableau/tableau-formula-translation.md	Tableau → ThoughtSpot formula and function mapping
../../shared/mappings/tableau/tableau-tml-rules.md	TML generation rules — critical invariants for valid import
../../shared/schemas/thoughtspot-table-tml.md	Table TML structure reference
../../shared/schemas/thoughtspot-model-tml.md	Model TML structure reference
../../shared/schemas/thoughtspot-sql-view-tml.md	SQL View TML structure — for custom SQL datasources
../../shared/schemas/thoughtspot-liveboard-tml.md	Liveboard TML structure reference
../../shared/schemas/thoughtspot-answer-tml.md	Answer/visualization TML structure
../../shared/schemas/thoughtspot-chart-types.md	Verified answer.chart.type enum (44 values) + analytical-intent → chart-type mapping
../ts-profile-thoughtspot/SKILL.md	ThoughtSpot auth setup
references/open-items.md	Known validation quirks and workarounds
references/coverage-matrix.md	Canonical mapped/unmapped construct matrix — cite in Audit mode (A4) and the migration report (Step 12)
references/liveboard-style-themes.md	Step 10.5 curated themes — brand tokens + per-chart viz_style color palettes

---

Prerequisites

• ThoughtSpot profile configured — run /ts-profile-thoughtspot if not
• ts CLI installed: pip install -e tools/ts-cli
• Tableau workbook file (.twb or .twbx) accessible on disk
• Tableau profile configured (optional) — run /ts-profile-tableau if migrating workbooks with published datasources (sqlproxy). Not needed for workbooks with direct connections.
• The source tables and their data already exist in a warehouse, and a ThoughtSpot connection exposes them. This skill creates ThoughtSpot logical objects (Table, Model, cohorts, Liveboard) over existing physical tables — it does not create warehouse tables or load/populate data. A ThoughtSpot table binds to a live connection that already surfaces the physical table and its columns (see Step 4.5 / thoughtspot-table-tml.md); if no such connection/table exists, set that up first (the data pipeline is out of scope). The skill may read the warehouse for confirmation (value formats, ranges, membership) — with your authorization — but never loads or modifies data.

---

CLI-first rule — no inline Python for TML operations

Every ThoughtSpot API call, TML generation, formula translation, and model import in this skill must go through a ts CLI command. Do not write inline Python scripts to export/merge/import TML, iterate over formula failures, or assemble model JSON.

Operation	Use this	NOT this
Translate formulas	ts tableau translate-formulas	Inline Python calling tableau_translate.py / the ts_cli/tableau/ package modules
Build/merge model + formulas	ts tableau build-model --existing-guid	Manual export → JSON merge → import loop
Create tables	ts tables create	Raw HTTP calls to the TML import endpoint
Export/import TML	ts tml export / ts tml import	Manual curl or HTTP library calls

If a CLI command fails or produces wrong results, fix the CLI (tools/ts-cli/) and re-run — do not work around it with manual scripting. The CLI encodes months of edge-case fixes (parameter conflict renaming, cross-reference resolution, iterative retry, double-aggregation detection) that inline scripts will miss.

---

Working principle — surface, recommend, resolve

Whenever the parse or generation hits a situation that has no clean 1:1 automatic translation or needs a judgement call — e.g. a cross-datasource blend, a join key that doesn't exist / spans two tables, a date stored as VARCHAR, bins (formula vs cohort), an ambiguous count column, a manual group (cohort vs if/then), an untranslatable formula, or a value-vs-data mismatch — do not silently drop it, guess, or merely flag it. Instead:

1. Surface it — tell the user plainly what was found and why it's not a straight translation.
2. Recommend — if there's a sound solution (or a small set of options), say which and why, with the trade-offs.
3. Resolve — with the user's go-ahead, do it (build the SQL view, prompt for the value, retype the column, create the cohort, etc.). Only fall back to omit-and-flag when no solution exists or the user declines.

Default to enabling the migration, not abandoning the hard parts. The per-step prompts and checkpoints below are how this principle is applied in practice.

Read the actual calculation — never infer from the name. A worksheet called "Highest Growth in past 5 years" tells you the intent, not the logic. Always inspect the real Tableau definition — the table-calc type (pcdf, pctd, running_*), the filters (Top-N, recent-N-years), the compute-using/partition, and the sort — and translate that. (Example: that title is really "top 5 sectors by FDI % change over a 6-year window" — a period comparison, not a raw growth of line.)

Placeholder charts when a full translation isn't possible. If a viz can't be fully reproduced, don't silently omit it — build a placeholder: a TABLE with the columns you can produce, and write a note in both the viz's answer.description and the Migration Summary tab that the chart is partial and needs review. A visible, labelled stub the user can finish beats a missing tile.

---

Step 0 — Overview

On skill invocation, display this plan before doing any work:

---

ts-convert-from-tableau — convert a Tableau workbook into ThoughtSpot TML objects, with optional dashboard-to-liveboard migration.

Modes

A Audit — analyse a TWB file (or multiple files) and report migration coverage. No ThoughtSpot auth required. No TMLs generated. Use this to assess feasibility before committing to a migration.

M Migrate — full conversion: parse, generate TMLs, validate, and import.

Enter A / M:

Migrate scope (ask right after M — see Step 1.5)

When the user picks M, immediately ask what to migrate — this decides which steps run:

1 Models + Liveboards — full flow (tables/models, then dashboards → liveboards). (default) 2 Tables + Models only — build the data layer; skip dashboards/liveboards (skip Steps 8–11; go model → Step 11.5 coverage → Step 12 report). 3 Liveboards only — the model(s) already exist in ThoughtSpot; skip table/model creation (skip Steps 4–7.5) and build liveboards on a user-selected existing model (see Step 1.5 model picker). Still parse the TWB for dashboards (Step 3). 4 Models only — tables already exist in ThoughtSpot; skip table creation but build model(s) with formulas. Reuse existing table GUIDs (Step 4 E/G path). (skip Steps 5a table TML generation, 8–11; run Steps 4→5b→5.5→6→7→7.5→11.5→12). 5 Tables only — generate and import table TMLs only; skip models and liveboards (run Steps 4→4.5→5a→6→12). Useful when tables need to be created/updated before a separate model migration pass.

Steps (Migrate mode)

1. Authenticate to ThoughtSpot .......................... auto 1.5 Choose migration scope (1–5) and pace (F/C) ...... you choose
2. Locate and extract the TWB file ...................... you provide path
3. Parse TWB XML — extract tables, columns, joins, calculated fields, blend relationships, table-calc addressing ............................ auto 3.5 Resolve published datasources (sqlproxy → API) ... auto/you choose [scope 1,2,4,5] 3.6 Confirm joins (present/suggest/range join option) . you confirm [scope 1,2,4]
4. Confirm source tables (reuse/GUID/create/search) ..... you choose [scope 1,2,4,5] 4.5 Select ThoughtSpot connection (create path only) .... you choose [scope 1,2,5]
5. Generate TML files ................................. auto [scope 1,2,4,5] 5a Table TMLs (+ sql_view) ......................... auto [scope 1,2,5] 5b Model TML + formula translation ................. auto [scope 1,2,4] 5.5 Confirm Spotter (AI search) enablement (default Y) .. you choose [scope 1,2,4]
6. Validate against ThoughtSpot (up to 10 fix cycles) .. auto [scope 1,2,4,5]
7. Review checkpoint + two-phase import: you confirm [scope 1,2,4] Phase 1: base model (tables, columns, joins, params — NO formulas) Phase 2: add formulas (GUID-pinned update, iterative error recovery) 7.5 Confirm the model is correct (test in Search/Spotter) you confirm [scope 1,2,4]
8. Migrate dashboards? + separate vs single-tabbed (2+) . you choose (skip → Step 12) [scope 1,3]
9. Parse dashboard layout and map to grid ............... auto [scope 1,3] 9d. Orphan worksheets (not on a dashboard) — add as tiles? you choose [scope 1,3] 10c. Choose charting library (Legacy default / Muze) ........ you choose [scope 1,3]
10. Generate liveboard TML (export model for params first) auto [scope 1,3] 10f. Add referenced parameters to the header? (default Y) . you choose [scope 1,3] 10g. Add a "Migration Summary" tab (migrated/decisions/omitted) auto [scope 1,3] 10.5 Pick a liveboard style (curated theme; default) ..... you choose [scope 1,3]
11. Import liveboard ..................................... you confirm [scope 1,3] 11.5 Formula coverage answers (every formula testable) ... auto [scope 1,2,4]
12. Migration report (outcomes + links + formula map) ... auto [scope 1,2,3,4,5] 12.5 Resume prompt — fix parked formulas? .............. you choose [scope 1,2,4; if parked]

Confirmation required: Steps 1.5, 3.6, 4.5, 5.5, 7, 7.5, 8, 9d, 11, 12.5 Auto-executed: Steps 1, 3, 5, 6, 9, 10, 11.5, 12 Scope 2 (Tables + Models) skips 8–11; runs 11.5 then 12. Scope 3 (LB only) skips 4–7.5; runs 1.5a model picker then 8–12. Scope 4 (Models only) skips 4.5, 5a, 8–11; runs 4(E/G), 5b, 6–7.5, 11.5, 12. Scope 5 (Tables only) skips 5b, 5.5–7.5, 8–11.5; runs 4, 4.5, 5a, 6, 12.

Efficiency — keep the migration fast

The flow is interactive, but most of the wall-clock cost is avoidable. Apply these:

• Batch independent prompts. Use a single multi-question prompt for decisions that don't depend on each other: mode + scope; the count-column + bin-style + cohort-handling decisions; theme + parameter-chips. Only serialize genuinely dependent questions (e.g. search-scope → connection name).
• Parse the TWB in one pass. Extract datasources, columns, calc fields, parameters, dashboards, zones, and table-calc addressing in a single script — not one Bash call per element.
• Read the model's real obj_id once, up front (Step 10-pre) — exporting the model once yields obj_id and parameters[].id and the resolved column names. This prevents the slow build→import→fail→delete→re-import liveboard cycle (see the obj_id rule in Step 7 / Step 10-pre).
• Don't fetch what you don't need. Skip ts connections list / ts connections get whenever the user names the connection or the tables are reused (Steps 4/4.5). Skip the whole model/table layer entirely in scope 3 (LB only).

Steps (Audit mode)

A1. Locate and extract TWB file(s) ...................... you provide path(s) A2. Parse TWB XML — same extraction as Step 3 .......... auto A3. Classify formulas into translation tiers ............ auto A4. Migration coverage report ........................... auto

No auth, no TML generation, no import. Supports multiple files in one run.

---

If Audit mode, proceed to Step A1. If Migrate mode, proceed to Step 1 (then Step 1.5 picks the migration scope, which gates the later steps).

---

Step A1 — Locate TWB File(s) (Audit Mode)

Ask: "Provide the path to a .twb or .twbx file, or a directory containing multiple workbooks."

If a directory is provided, find all .twb and .twbx files recursively. For each .twbx, extract to a temp directory to access the inner .twb.

Save the list of TWB paths. Process each file through Steps A2–A4 independently.

---

Step A2 — Parse TWB XML (Audit Mode)

Run the same extraction as Step 3 (3a through 3e) on each TWB file. Do NOT skip any datasource type. For extracts, resolve the underlying source (Step 3b) and report it as migratable via that source; mark as "Extract — no underlying source" only when none resolves.

---

Step A3 — Classify Formulas (Audit Mode)

MANDATORY (I7) — before classifying any calculated field as untranslatable, open ../../shared/mappings/tableau/tableau-formula-translation.md and check its full function table and pass-through section. Do not decide from syntax alone.

Run the classifier — it shares the migrate-mode translation verdict, so the audit cannot over- or under-promise coverage:

ts tableau classify-formulas --input /tmp/ts_tableau_mig/{workbook_name}_parsed.json --output /tmp/ts_tableau_mig/audit/{workbook_name}_classification.json

The classifier works per datasource — each datasource becomes its own model in migration, and the same calc name can carry a different expression in each datasource, so it must be tiered against its own. For a parsed-workbook input the output is { "datasources": [ {"name", "formulas", "tier_counts", "translate_stats"}, … ], "tier_counts": <sum across datasources> }. Each formulas[] entry has tier, reason, level, and complexity. Report per datasource (Step A4's per-datasource breakdown) and use the top-level tier_counts for the workbook total. (Pass --datasource "<name>" to limit to one; a bare-list input — e.g. Step 5b's translate-formulas file — instead yields a flat {formulas, tier_counts, translate_stats}.)

Translatable tiers: native, lod, cumulative, moving, pass_through, row_offset_native, parameter_ref. Untranslatable tiers: untranslatable, row_offset_ambiguous, geospatial, circular, orphan, parameter_query.

The table below maps those tiers to the human-readable categories used in the report — kept as reference/documentation now, not as executed classification logic:

Tier	Description	Examples
Native / Set	Direct ThoughtSpot mapping exists	IF/THEN, IFNULL, DATEDIFF, LEFT, ABS, ROUND, IIF; bins (class='bin') → floor([x]/size)*size or BIN_BASED cohort; manual groups (class='categorical-bin', incl. fields named "… clusters") → GROUP_BASED cohort; Number of Records/row counts → count([column]) (prompt for the column; default the primary key); static sets (<group> with union/member) → GROUP_BASED column-set cohort — incl. ones anchored on a formula column, with a %null% member (via EQ {Null}), or an except member-list (via NE) (Phase 2a); Top-N/Bottom-N sets (function='end') → query set (cohort_type: ADVANCED, COLUMN_BASED) via a rank formula + parameter-filter formula (Phase 2b); condition-based sets (function='filter') → query set with aggregate condition formula (Phase 2c); member-list intersect → GROUP_BASED cohort of common members (Phase 2c); all-except-Top-N → query set with inverted rank filter (Phase 2c); computed set operations (intersect/except of mixed types) → multi-formula query set (Phase 2c)
LOD	LOD expression → group_aggregate()	{FIXED dim : SUM(col)}; TOTAL(SUM(x)) / percent-of-total → group_aggregate(..., {}, query_filters())
Cumulative	Running calculation → cumulative_*()	RUNNING_SUM, RUNNING_AVG
Moving	Window table calc → moving_*()	WINDOW_SUM, WINDOW_AVG (when sort attr determinable)
Pass-through	Valid SQL but no native function → sql_*_aggregate_op()	Partitioned RANK, DENSE_RANK, WINDOW_* without sort context
Partial / Unmapped (sets)	Tableau set construct with no current ThoughtSpot equivalent — logged as deferred, never mis-translated	set controls (level-members only, no fixed members) → no set object, surface as a liveboard filter; set actions (<action>) → no equivalent
Row-offset (native)	Table calc with recoverable intent → native TS function	INDEX() <= N (Top-N filter → rank() + query set); INDEX() display → rank(); LOOKUP(-N) → moving_sum(N,-N); LOOKUP(+N) → moving_sum(-N,N); LOOKUP(agg, FIRST()) → first_value(); LOOKUP(agg, LAST()) → last_value(); bare FIRST()/LAST() standalone → omit + log (returns offset, not value) — all native mappings use native TS functions, work with all column types
Row-offset (pass-through)	SIZE() only → answer-level sql_int_aggregate_op("COUNT(*) OVER()")	SIZE() — only row-offset that still requires SQL pass-through ⚑ flag PT1
Untranslatable	No ThoughtSpot equivalent — will be omitted	INDEX/LOOKUP/FIRST/LAST/SIZE when addressing is ambiguous (CellInPane, multi-dim Table, or no shelf sort); PREVIOUS_VALUE (true recursion — not the string-aggregation technique); true k-means clustering (the analytics-engine "Clusters" calc — not categorical-bin); geospatial — full 13-function set (MAKEPOINT, MAKELINE, BUFFER, OUTLINE, DISTANCE, AREA, LENGTH, INTERSECTS, SHAPETYPE, DIFFERENCE, INTERSECTION, SYMDIFFERENCE, VALIDATE) — decompose MAKEPOINT lat/lon args to individual attribute columns, omit the spatial formula (see tableau-formula-translation.md Geospatial Policy); embedded-RLS user attributes (USERATTRIBUTE, USERATTRIBUTEINCLUDES) — rejected at translate time, see BL-071
Parameter ref (auto)	References a Tableau parameter with static list/range — parameter auto-created in model	[Parameters].[Currency] where Currency has <member> values
Parameter ref (query)	References a Tableau parameter with SQL-lookup list — queryable at migration time	SQL-populated parameter lists (needs connection)

---

Step A4 — Migration Coverage Report (Audit Mode)

For each TWB file, produce a coverage report. If multiple files were audited, also produce a combined summary at the end.

Source the numbers from classification.json (Step A3's ts tableau classify-formulas output) — do not hand-tally tiers. The output is per datasource: iterate classification.json's datasources[] for the per-datasource breakdown (each carries its own formulas[], tier_counts, and translate_stats — where total == translated + skipped for that model), and use the top-level tier_counts for the workbook total. The per-formula rows (Row-offset detail, Excluded Formulas, "Needing Review") come from each datasources[].formulas[] entry's tier/level/complexity/reason fields.

Per-file report:

Audit: {workbook_name}
══════════════════════════════════════════════════════

  Datasources:          {N} total
    Live:               {N}
    Extract:            {N} (skipped in migration)
    Published (sqlproxy): {N}

  Physical tables:      {N}
  Custom SQL relations: {N} → will generate sql_view TMLs
  Joins:                {N}

  Calculated fields:    {N} total
  ┌──────────────────────────────────────────────────────┐
  │ Tier                    Count    %     Examples      │
  ├──────────────────────────────────────────────────────┤
  │ Native                  {N}     {%}   IF, DATEDIFF  │
  │ LOD → group_agg         {N}     {%}   {FIXED ...}   │
  │ Cumulative              {N}     {%}   RUNNING_SUM   │
  │ Moving                  {N}     {%}   WINDOW_SUM    │
  │ Pass-through            {N}     {%}   DENSE_RANK    │
  │ Row-offset (native)     {N}     {%}   LAG→moving_sum│
  │ Row-offset (pass-thru)  {N}     {%}   SIZE→COUNT(*) │
  │ Parameter ref (auto)    {N}     {%}   static list   │
  │ Parameter ref (query)   {N}     {%}   SQL lookup    │
  │ Geospatial (omit+log)   {N}     {%}   MAKEPOINT     │
  │ Untranslatable          {N}     {%}   INDEX(ambig)  │
  └──────────────────────────────────────────────────────┘

  Cross-reference depth (formula-to-formula dependencies):
  ┌──────────────────────────────────────────────────────┐
  │ Depth                   Count    %     Note          │
  ├──────────────────────────────────────────────────────┤
  │ Level 0 (self-contained) {N}    {%}   Translate directly │
  │ Level 1 (refs Level 0)   {N}    {%}   Inline from Level 0 │
  │ Level 2+ (deep chains)   {N}    {%}   Multi-level inlining │
  │ Circular dependencies    {N}    {%}   Cannot resolve │
  └──────────────────────────────────────────────────────┘

  Formula complexity (effort estimate):
  ┌──────────────────────────────────────────────────────┐
  │ Complexity              Count    %     Criteria      │
  ├──────────────────────────────────────────────────────┤
  │ Simple (1–2)            {N}     {%}   ≤1 function, no cross-refs, no nesting │
  │ Medium (3–5)            {N}     {%}   2–3 functions or 1 cross-ref or 1 nesting level │
  │ Complex (6+)            {N}     {%}   4+ functions, 2+ cross-refs, or deep nesting │
  └──────────────────────────────────────────────────────┘
  Score = nesting_depth + cross_ref_count + function_count (each function/operator = 1)

  Effective migration coverage:
    Syntax-level:            {N}/{total} ({%}%) — based on tier classification only
    After orphan exclusion:  {N}/{total} ({%}%) — minus {N} orphan inherited calcs
    After dep resolution:    {N}/{total} ({%}%) — minus {N} circular, {N} unresolvable cross-refs
    ──────────────────────
    Realistic estimate:      {N}/{total} ({%}%)

  ⚠ The tier classification reports SYNTAX-LEVEL translatability. The realistic
  estimate accounts for orphan calcs (Step 3g), circular dependencies, and
  unresolvable cross-references. A formula classified "Native" may still not
  migrate if it depends on an orphan or circular chain.

  Tableau Sets (top-level <group> elements — separate from calculated fields):
  ┌──────────────────────────────────────────────────────┐
  │ Set tier                Count    Notes               │
  ├──────────────────────────────────────────────────────┤
  │ Native / column set     {N}      static + member-intersect → GROUP_BASED cohort (2a/2c) │
  │ Query set               {N}      Top-N, condition, all-except-Top-N, mixed ops → ADVANCED (2b/2c) │
  │ Partial / deferred      {N}      set controls + set actions (no equivalent)     │
  └──────────────────────────────────────────────────────┘

  Parameters:           {N} total ({N} static, {N} SQL-lookup — query at migration)
  Dashboards:           {N} (optional liveboard migration)

  ──────────────────────────────────────────────────
  Migration coverage:   {(all except untranslatable-ambiguous) / total}%
                         (all parameters auto-created — static or queried)
  Untranslatable:       {N} formula(s) — will be omitted
  Geospatial:           {N} formula(s) — spatial funcs omitted; lat/lon cols migrated as attributes
  Deferred sets:        {N} (set controls/actions — flagged for manual creation)
  SQL-lookup params:    {N} — need warehouse connection at migration time
  Pass-through formulas (DENSE_RANK, SIZE, etc.) require SQL Passthrough Functions enabled.
  Row-offset native formulas (LAG, LEAD, LOOKUP(agg,FIRST/LAST), INDEX) use native TS functions — no pass-through needed.
  Bare FIRST()/LAST() standalone → omitted (returns offset, not value — no TS equivalent).

  Migration effort estimate:
    Formula translation:  deterministic (CLI pipeline, no LLM calls)
    Phase 1 import:       ~1-2 minutes (tables + base model)
    Phase 2 import:       ~{N} retry cycles expected
      Simple formulas:    {N} — likely import on first try
      Medium formulas:    {N} — may need 1-2 retries
      Complex formulas:   {N} — expect structural fixes
    Estimated total:      {M}-{N} minutes (model only, excludes liveboard)

  Orphan inherited calcs:  {N} formula(s) referencing missing tables
  ┌──────────────────────────────────────────────────────────────────────────┐
  │ These calcs were inherited from a parent/copied datasource and          │
  │ reference tables not present in this datasource. They are               │
  │ non-functional in Tableau and will be excluded from migration.          │
  │                                                                         │
  │ Missing tables: {table1}, {table2}, …                                   │
  │                                                                         │
  │  #   Formula Name              Missing Table Reference                  │
  │  1   {name}                    {TABLE_NAME}                             │
  │  …                                                                      │
  │                                                                         │
  │ (Includes {N} transitive orphans — depend on a direct orphan)           │
  └──────────────────────────────────────────────────────────────────────────┘

  ⚠ Formulas Needing Review:  {N} formula(s)
  ┌──────────────────────────────────────────────────────────────────────────┐
  │  #   Formula Name              Category           What to Verify        │
  ├──────────────────────────────────────────────────────────────────────────┤
  │  1   {name}                    No-keyword LOD      Test with/without    │
  │                                                    search filters       │
  │  2   {name}                    Blend-context        Compare totals      │
  │  3   {name}                    Pass-through SQL     SQL Passthrough on? │
  │  4   {name}                    ifnull stripped      Verify null display │
  │  5   {name}                    sum_if rewrite       Verify aggregation  │
  │  …                                                                      │
  ├──────────────────────────────────────────────────────────────────────────┤
  │ Categories:                                                             │
  │  No-keyword LOD   {AGG([col])} → group_aggregate — filter context may  │
  │                   differ (Tableau: after dim filters, before table-calc)│
  │  Blend-context    Secondary datasource ref — post-agg blend vs row join│
  │  Pass-through SQL sql_*_aggregate_op — requires cluster setting enabled│
  │  ifnull stripped  ifnull(measure,0) removed — TS handles NULLs natively│
  │  sum_if rewrite   sum(if…then…else 0) → sum_if() — semantically equiv │
  └──────────────────────────────────────────────────────────────────────────┘

  Data Blending — post-aggregation semantics:
  ┌──────────────────────────────────────────────────────────────────────────┐
  │ ⚠ Tableau blends aggregate the secondary datasource independently     │
  │ before joining. ThoughtSpot model joins operate at row level.          │
  │ If both sides are fact tables at different grains, aggregation         │
  │ results may differ.                                                    │
  ├──────────────────────────────────────────────────────────────────────────┤
  │ Primary DS          Secondary DS       Link Cols    Risk               │
  ├──────────────────────────────────────────────────────────────────────────┤
  │ {ds_caption}        {ds_caption}       {col1, …}   {HIGH/MED/LOW}     │
  │ …                                                                      │
  ├──────────────────────────────────────────────────────────────────────────┤
  │ Risk: HIGH = both sides have measures (fact×fact — aggregation may     │
  │              diverge)                                                   │
  │       MED  = secondary has measures but likely reference table         │
  │       LOW  = secondary is dimension-only                               │
  └──────────────────────────────────────────────────────────────────────────┘
  Blended datasources:  {N} of {total} — will merge into single model(s)
  Blend relationships:  {M} total
  Star topologies:      {S} (1 primary → 2+ secondaries)
  HIGH-risk blends:     {N} — verify aggregation results post-migration
  ──────────────────────────────────────────────────

Migration coverage includes everything except Untranslatable. All parameter types are auto-migratable: static params are created directly in the model TML; SQL-lookup params are populated by querying the warehouse at migration time. The formula reference [Parameters].[Name] is rewritten to [Name] in both cases. Cite references/coverage-matrix.md as the canonical source when classifying a construct as mapped/unmapped or explaining a limitation.

If any formulas are classified as Row-offset, list them:

  Row-offset formulas (translated via decision tree):
    - {formula_name}: {tier} — {tableau_expr} → {ts_expr or "omit (ambiguous)"}
    - ...

Excluded formulas — every formula that will NOT be migrated, grouped by root cause:

  Excluded Formulas: {N} total
  ┌──────────────────────────────────────────────────────────────────────────┐
  │ Root Cause               Count  Potential Resolution                    │
  ├──────────────────────────────────────────────────────────────────────────┤
  │ Untranslatable function  {N}    No TS equivalent — SQL view or UDF      │
  │ Missing table in model   {N}    Add source tables, then create formula  │
  │ Circular dependency      {N}    Break cycle by inlining one formula     │
  │ Complex date arithmetic  {N}    Rewrite with TS date funcs or warehouse │
  │ Geospatial function      {N}    Spatial not supported — lat/lon as attr │
  └──────────────────────────────────────────────────────────────────────────┘

  Per-formula detail:
    {Root cause category} ({N}):
    - {formula_name}: {tableau_expr} — {what the user can do}
    - ...

If any SQL-lookup parameters exist, note them:

  SQL-lookup parameters ({count} — populated from warehouse at migration time):
    - {param_name}: query/column reference from TWB
    - ...
  Values are a point-in-time snapshot. Consider /ts-recipe-parameter-sync for
  ongoing refresh.

If any formulas are classified as Pass-through, list them with the generated expression:

  Pass-through formulas (require SQL Passthrough Functions enabled):
    - {formula_name}: sql_{type}_aggregate_op("...", ...)
    - ...

Write the report to /tmp/ts_tableau_mig/audit/{workbook_name}_audit.md and display it inline.

Per-datasource breakdown (when a workbook has multiple datasources):

Report coverage per datasource, not just workbook-wide. Different datasources within the same workbook can have very different effective migration rates (e.g. a full datasource at 73% vs a copied subset at 54%). A combined number hides this.

  Per-datasource coverage:
  ┌──────────────────────────────────────────────────────────────────────────┐
  │ Datasource              Tables  Calcs  Orphans  Realistic Coverage     │
  ├──────────────────────────────────────────────────────────────────────────┤
  │ {ds_name_1}             {N}     {N}    {N}      {N}/{N} ({%}%)         │
  │ {ds_name_2} (copy)      {N}     {N}    {N}      {N}/{N} ({%}%)         │
  └──────────────────────────────────────────────────────────────────────────┘

Combined summary (multiple files):

Audit Summary: {N} workbook(s)
══════════════════════════════════════════════════════

  Workbook                          Tables  Calcs  Orphans  Coverage
  ─────────────────────────────────────────────────────────────────────
  {workbook_1}                      {N}     {N}    {N}      {%}%
  {workbook_2}                      {N}     {N}    {N}      {%}%
  ...
  ─────────────────────────────────────────────────────────────────────
  Total                             {N}     {N}    {N}      {%}%

  Coverage = realistic estimate (after orphan + circular + cross-ref exclusion)

After the audit, exit cleanly. Do NOT proceed to Migrate mode steps.

---

Step 1 — Authenticate

Run ts profiles list to discover available ThoughtSpot profiles. If no profiles exist, run /ts-profile-thoughtspot to create one. If multiple profiles exist, display a numbered menu and ask the user to choose. If only one profile, use it automatically.

ts profiles list

Then verify the chosen profile:

source ~/.zshenv && ts auth whoami --profile "{profile_name}"

Save {base_url} and {profile_name} for all subsequent steps.

---

Step 1.5 — Choose Migration Scope

Right after auth, ask what to migrate (this can be batched with the profile choice when there are multiple profiles). The answer gates which later steps run:

What should I migrate?
  1  Models + Liveboards — build/reuse tables & models, then dashboards → liveboards  (default)
  2  Tables + Models only — build the data layer; skip dashboards/liveboards
  3  Liveboards only — models already exist; skip table/model creation and build
                       liveboards on an existing model I help you pick
  4  Models only — tables already exist in ThoughtSpot; skip table creation,
                   build model(s) referencing existing tables
  5  Tables only — generate and import table TMLs; skip models and liveboards

Migration pace?  (scopes 1, 2, 4 only — omit for scopes 3 and 5)
  F  Fast — import formulas, park any failures, move on  (default)
  C  Complete — after import, attempt to fix each failure (slower)

Enter scope (1-5) and pace (F/C):

Apply the scope:

Scope	Runs	Skips
1 Models + Liveboards	all steps	—
2 Tables + Models only	2–7.5, then 11.5 (coverage), 12	8–11 (dashboards/liveboards)
3 Liveboards only	1.5a model picker, 2–3 (parse, dashboards), 8–12	4–7.5 (table/model creation)
4 Models only	2–3 (parse), 4 (E/G to find existing tables), 5b (model TML + formulas), 5.5, 6, 7, 7.5, 11.5, 12	4.5 (connection — tables already bound), 5a (table TMLs), 8–11 (liveboards)
5 Tables only	2–3 (parse), 4, 4.5, 5a (table TMLs), 6, 12	5b (model), 5.5–7.5, 8–11

Save the pace as {migration_pace} (F or C). Default F if the user enters only a scope number or skips the pace question. For scopes 3 and 5, {migration_pace} is always F (no formula imports, so the pace has no effect).

For scope 2, after Step 7.5 jump straight to Step 11.5 then Step 12.

For scope 3, there is no model to build — the user selects an existing model, and the liveboard tiles reference its columns/formulas. Run Step 1.5a below to pick it, then parse the TWB (Steps 2–3) and continue at Step 8. (Step 9b maps each worksheet's shelves to the chosen model's columns by display name — surface any field that has no matching column rather than guessing.)

For scope 4, tables already exist in ThoughtSpot — the user provides GUIDs or searches for them (Step 4, E or G path). No connection selection or table TML generation needed. The model TML references existing tables by GUID. This is the common path for consultant/remote migrations where tables were loaded separately (e.g. via /ts-load-source-data or manual warehouse provisioning + ts tables create). After Step 7.5, jump to Step 11.5 then Step 12.

For scope 5, only table TMLs are generated and imported. No model, no formulas, no liveboard. Useful for a phased migration where tables are set up first, then models are created in a second pass (scope 4).

Step 1.5a — Pick an existing model (scope 3 only)

A ThoughtSpot Model is a worksheetVersion: V2 logical table — there is no MODEL subtype in metadata search. Find models with --subtype WORKSHEET (which returns worksheets and models) and keep only those whose metadata_header.worksheetVersion == "V2".

Prompt how to identify the model — don't list every model by default (the full list is slow on a large instance). Mirror the connection picker (Step 4.5):

How would you like to choose the model?
  G  GUID         — paste the model's GUID; I'll fetch it directly      (fastest)
  N  Name it      — type the exact model name
  F  Filter       — give a partial string; I'll list matching models
  L  List all     — show every model and pick by number   (slow — scans all worksheets)

Enter G / N / F / L:

Resolve the choice:

• G (GUID) — fetch directly and confirm it's a model:

  source ~/.zshenv && ts metadata search --guid {model_guid} --profile {profile_name}
  

  Verify metadata_type == "LOGICAL_TABLE" and metadata_header.worksheetVersion == "V2". If V1 (a classic worksheet) or not found, say so and re-ask.
• N (name it) — exact-name search, filter to V2:

  source ~/.zshenv && ts metadata search --subtype WORKSHEET --name "{model_name}" --profile {profile_name}
  

  Exactly one V2 match → use it; none/ambiguous → show closest and re-ask.
• F (filter) — --name "%{partial}%", keep V2 matches, show a short numbered list (name, obj_id, guid) and pick from it.
• L (list all) — warn it's slow, then --subtype WORKSHEET --all, keep V2, show the numbered list. Only use when the user can't name/filter.

# F / L pattern (filter applied client-side to V2 only)
source ~/.zshenv && ts metadata search --subtype WORKSHEET --name "%{partial}%" --profile {profile_name}

From the chosen model capture and save: {model_guid} (metadata_id), {model_obj_id} (metadata_obj_id — the real obj_id; see the obj_id rule in Step 7), and {model_name} (metadata_name). Then export it once (Step 10-pre) to read its columns, formulas, and parameters[].id for building liveboard tiles. Confirm the picked model with the user before proceeding:

Using existing model: {model_name}
  {base_url}/#/data/tables/{model_guid}
  Columns: {n}   Formulas: {f}   Parameters: {names}
Build liveboards on this model? (yes / pick another)

---

Step 2 — Extract TWB File

Ask for the file path if not yet provided.

If the file ends in .twbx (a ZIP archive), extract it:

mkdir -p /tmp/ts_tableau_mig && unzip -o "{twbx_path}" -d /tmp/ts_tableau_mig/

Then find the .twb inside:

find /tmp/ts_tableau_mig -name "*.twb" | head -1

Save the resolved .twb path as {twb_path}.

---

Step 3 — Parse TWB XML

Run the parser and read its JSON — do NOT hand-parse the XML:

ts tableau parse "{twb_path}" --output /tmp/ts_tableau_mig/{workbook_name}_parsed.json

The JSON contains datasources[] (each with tables, columns, joins, calculated_fields, calc_map, col_table_map, orphan_calcs), parameters, param_map, blends, and table_calc_addressing. All subsequent steps read these fields instead of re-deriving them.

The parse output (from the ts tableau parse call above) contains the following, extracted from the TWB's XML structure:

3a. Workbook name

Take from the filename (strip .twb). Save as {workbook_name}.

3b. For each <datasource> element (skip those named Parameters)

Each datasource is processed independently for extraction (Steps 3b–3d). Datasources are merged into a single model only when they are connected by blend relationships (detected in Step 3e). Even when datasources share tables or point at the same database, they are NOT merged unless a <datasource-relationship> explicitly links them. See Step 5b "Blend-aware model grouping" for the merge procedure.

Datasource type detection:

• If the datasource contains <connection class="sqlproxy">, it is a Published Datasource (hosted on Tableau Server). The table name resolves to connection.get('dbname'), not the literal [sqlproxy].
• If the datasource contains <extract>, do not blindly skip it. An extract is a local snapshot, but it almost always wraps an underlying connection that names a real table — a file source (textscan/CSV, excel-direct), a database, etc. What matters for migration is that underlying source, because that's what gets queried in the warehouse. Look past the <extract>/hyper connection to the real one:
  • The relation has two parents in <metadata-records> — the live source (e.g. [Amazon Sales data.csv]) and [Extract]. Use the live-source relation; ignore the [Extract] relation. The physical table name comes from the live source (mapped to its warehouse table per Step 4.5).
  • Only treat a datasource as truly skippable when there is no resolvable underlying connection (a pure Tableau-authored extract with no source) — and say so in the report.
  • File-based sources (CSV/Excel) imply the data was loaded into the warehouse out of band; bind the table to the connection that now exposes it (Step 4/4.5).
• Otherwise, it is a Live datasource — proceed with extraction.

Non-warehouse sources — explicit unsupported policy: The following Tableau connection classes are NOT warehouse-bound and cannot be mapped to a ThoughtSpot connection: cloudfile:googledrive-excel-direct, google-sheets, ogrdirect (spatial/OGR), webdata-direct (web data connector), CustomMapbox. When any of these appear as a datasource's connection class, do NOT assume a warehouse table exists. Instead:

1. Log: "Datasource '<name>' uses a non-warehouse source (<class>) — cannot map to a ThoughtSpot connection. Skipped; data must be loaded into a warehouse first."
2. Skip the datasource entirely (do not generate table or model TML for it).
3. Surface in the audit report under a "Skipped sources" section.

Redshift and Postgres dialect notes: When <connection class="redshift"> or <connection class="postgres"> is detected, pass-through SQL (sql_*_op) formulas should use the corresponding dialect syntax. Key differences from Snowflake:

• String concatenation: || (same as Snowflake)
• Date truncation: date_trunc('month', col) (same syntax, both dialects)
• LISTAGG → Redshift: LISTAGG(col, ',') WITHIN GROUP (ORDER BY col); Postgres: string_agg(col, ',' ORDER BY col)
• Type casting: Redshift uses ::type; Postgres uses CAST(x AS type) or ::type

No other mapping changes are needed — the Tableau-to-ThoughtSpot formula translation is warehouse-agnostic (ThoughtSpot formulas are the target, not SQL). The dialect only matters for sql_*_op pass-through functions.

Relation wrapper handling: TWB XML wraps <relation> elements in one of three structures. Check in order:

1. _.fcp.ObjectModelEncapsulateLegacy.false...relation tag
2. _.fcp.ObjectModelEncapsulateLegacy.true...relation tag
3. <relation> directly under <connection class='federated'> (fallback)

All three contain the same child elements — the wrapper determines where to look.

For each datasource, extract:

Physical tables — <relation> elements of type="table":

• name attribute = table alias used in joins
• table attribute = fully-qualified physical table name — may be [DB].[SCHEMA].[TABLE] format; strip brackets and split on . to extract db, schema, and table components
• For Published Datasources (sqlproxy): if table name is [sqlproxy], use connection.get('dbname') instead

Custom SQL relations — <relation> elements of type="text":

• These contain raw SQL in the element text content — do NOT try to extract a table name
• Flag the relation as source_type: "custom-sql" and save the full SQL text
• Refactor the SQL: replace << with <, >> with >, == with = (XML encoding artifacts from the TWB)
• These will generate a sql_view: TML instead of a table: TML (see Step 5c)
• Extract column names from the SQL SELECT clause aliases for column mapping

Joins — <relation> elements of type="join":

• join attribute = join type (inner | left | right | full)
• <clause> child = join condition (decode HTML entities: "→", &→&, <→<, >→>)
• Extract left and right table references from the clause

Physical columns — from <metadata-records> → <metadata-record class="column">:

• local-name = column identifier
• remote-name = physical column name in the database (use for db_column_name)
• local-type = Tableau data type
• parent-name = which table this column belongs to
• Also extract from <column> elements WITHOUT a <calculation> child: name (strip brackets), datatype, role (dimension/measure), caption (display name)

Calculated fields — <column> elements WITH a <calculation class="tableau"> child:

• Skip columns where param-domain-type is list or range — these are Tableau parameters, not calculated fields
• caption or name = display name
• calculation formula attribute = Tableau expression (decode HTML entities)
• datatype attribute
• Build a cross-reference map: Tableau internal names ([Calculation_1234567890]) → display names. Calculated fields reference each other by internal ID in the TWB XML, not by display name — resolve these references before translating formulas.

Parameters — <datasource name="Parameters"> children:

• For each <column> with param-domain-type attribute:
  • caption = display name (used as ThoughtSpot parameter name)
  • datatype = string | integer | real | date | boolean
  • param-domain-type = list | range | any
  • value attribute or calculation.formula = default value
  • <member value="..."> children = list values (when param-domain-type="list")
  • <range min="..." max="..."> child = range bounds (when param-domain-type="range")
• Save parameter definitions — these generate model.parameters[] in Step 5b
• SQL-lookup parameters (where the list values come from a database query rather than static <member> elements): save the query/column reference — at migration time (Step 5b), query the warehouse to populate list_config.list_choice[] with current values. In audit mode (no connection), flag as "requires connection"

Save the parsed structure internally. Announce a summary:

Parsed {workbook_name}: {N} datasource(s), {N} physical table(s), {N} calculated field(s), {N} join(s), {N} dashboard(s)

3c. Topological sort of calculated fields

Some calculated fields reference other calculated fields. Sort them so that fields with no formula-dependencies come first (Level 0), then Level 1, etc. This determines the order they must appear in the model TML formulas section.

Resolve all internal Tableau cross-references ([Calculation_\d+] → display name) before sorting. The topological sort must use display names, not internal IDs.

3d. Dashboard metadata (for Step 8 decision)

Count <dashboard> elements in the TWB. Save the count and names — this is shown in Step 8 when asking whether to migrate dashboards.

3e. Extract blend relationships (data blending)

Parse the <datasource-relationships> element at the workbook root (child of <workbook>). If absent, no blending is used — skip this step.

Build the blend graph:

The parse-JSON blends field (Step 3) is this graph — {source_ds_caption: [{target_ds, column_mappings}]}, with federated IDs already resolved to datasource captions (matching each datasource's name from Step 3b). Treat blend_graph in the rest of this document as shorthand for that field.

What to log:

• Number of blend relationships found
• For each: primary datasource → secondary datasource, with linking columns listed

Date-grain linking columns:

When a <column-instance> has a derivation other than "None" (e.g. "Month", "Month-Trunc", "Year", "Year-Trunc"), the blend links at a specific time grain. For the ThoughtSpot model join, the physical date column is used directly — ThoughtSpot's date bucketing at query time handles the grain alignment.

However, if the source and target columns are physically different date columns with different native grains (e.g. source has daily Order Date, target has monthly Month of Order Date that is already pre-truncated), the join requires a date-truncation formula or SQL View to materialize the matching grain.

Resolution strategy:

1. If both columns are date/datetime type and the derivation indicates a truncation (Month-Trunc, Year-Trunc), emit a model formula: date_trunc ( 'month' , [TABLE::Order Date] ) and use that formula as the join key via a SQL View (the formula can't be a direct join key in model TML).
2. Surface the grain mismatch to the user in the review checkpoint with a recommendation:
   • "Blend links Order Date (daily) to Month of Order Date (monthly) at month grain. Recommend: create a SQL View with DATE_TRUNC('MONTH', ORDER_DATE) AS ORDER_MONTH and join on ORDER_MONTH = MONTH_OF_ORDER_DATE."
3. If both columns are the same physical type and grain, use them directly in the join on clause — no materialization needed.

No model merging happens here — this step only extracts the relationships. Model merging happens in Step 5b.

---

Step 3f — Table-calc addressing extraction

For every datasource, scan <column> elements that have a <calculation class='tableau'> child containing a <table-calc> element. This is the column-level addressing map — read it from the parse-JSON table_calc_addressing.column_level field (Step 3): keyed by calc internal ID (e.g. [Calculation_953355781789577216]), each entry has ordering_type (Rows | Columns | Table | CellInPane | Field), ordering_field, order_fields (list), quick_calc_type (PctTotal | PctDiff | Difference | PctRank | None), and address_offset (int or None).

Each <worksheet>'s <column-instance> elements can carry their own <table-calc> — these are view-level overrides that take precedence over the column-level definition for that worksheet. Read them from table_calc_addressing.ws_overrides (same entry shape, keyed by worksheet name then calc ID).

Resolution order when translating a table-calc formula used on worksheet W:

1. Check table_calc_addressing.ws_overrides[W][calc_id] — view-level override
2. Fall back to table_calc_addressing.column_level[calc_id] — column-level definition
3. If neither exists, treat as ordering_type='Rows' (Tableau default)

---

Step 3g — Orphan calc detection (copied datasources)

When a Tableau datasource is a copy of another (common with published datasource clones), it inherits all calculated fields from the original — including ones that reference tables no longer present in the copy. These orphan calcs are non-functional in Tableau and will fail at ThoughtSpot import.

Each datasource's orphan_calcs (parse-JSON field, Step 3) is this list — captions of calcs that directly reference a table missing from the datasource, plus calcs that transitively depend on a direct orphan.

What to log (if any orphans found):

⚠ Datasource {name}: {N} orphan calc(s) reference missing tables: {table1, table2, …} These are non-functional inherited fields and will be excluded from migration.

In migrate mode: surface the orphan count and the missing tables before proceeding. Ask the user to confirm exclusion — in rare cases they may want to add the missing tables to the connection and model instead:

{N} calculated fields reference tables not in this datasource ({table1, table2, …}). These appear to be inherited from a parent datasource and are non-functional. E — Exclude them (default, recommended) A — Add the missing tables to the model (you'll need to confirm they exist in Step 4)

If the user chooses A, add the missing tables to the datasource's table list and re-run Step 4 table confirmation for the new tables only. Remove the affected calcs from orphan_calcs so they enter the translation pipeline.

In audit mode: no prompt — just report the orphan count in the A4 audit report.

---

Step 3.5 — Resolve Published Datasources (sqlproxy)

Runs only if Step 3 detected one or more datasources with <connection class="sqlproxy">. Skipped entirely if all datasources have direct warehouse connections.

When a Tableau workbook references a published datasource on Tableau Server/Cloud, the TWB XML contains <connection class="sqlproxy"> with a dbname attribute naming the published datasource.

What the TWB DOES contain for sqlproxy datasources (extract these in Step 3b regardless of API access — they are <column> elements directly under the <datasource> element):

• All calculated fields with full Tableau formula text (same as direct-connection datasources)
• Column definitions with captions, datatypes, and roles
• Metadata records (local-name, remote-name, local-type, parent) — often complete enough for column mapping

What the TWB does NOT contain (it lives only in the published datasource's .tds):

• The physical table structure (table names, joins, db/schema/table paths)
• The connection details (database, schema) that link to the warehouse

This means formula extraction and translation work without the physical model — the .tds adds physical table resolution and join definitions, not the formulas.

Where the physical model actually is — and how to get it. The join/table structure is not returned by the field API. ts tableau datasource --fields (VizQL read-metadata) returns columns/calcs only, not tables or joins. The full physical model (tables, joins, custom SQL) lives in the published datasource's .tds. Two ways to obtain and parse it:

• Download it (needs Tableau access): ts tableau download {datasource_id} fetches the .tdsx; the .tds inside carries the model.
• Be supplied it: the user provides the .tds/.tdsx alongside the .twb.

Then ts tableau parse accepts a .tds/.tdsx directly (ts-cli ≥ 0.38.0) — its root is the <datasource>, and parse extracts its tables/joins/columns/calcs just like a workbook datasource. Feed that to build-model GENERATE mode and it builds the multi-table model automatically — no hand-assembly (see Step 5b "Multi-query datasources"). Without the .tds (only the .twb, no Tableau access), fall back to the hand-built multi-table base in Step 5b.

Flow

ASK before querying the Tableau API. The user may be a consultant conducting a remote migration without access to the customer's Tableau Server. Do NOT attempt any API call before asking — a failed API call wastes 30–60 seconds and confuses the flow.

1. Prompt — always, before any API call:

   Found {N} published datasource(s) hosted on Tableau Server:
     - {ds_caption_1} ({M} columns, {C} calculated fields extracted from TWB)
     - {ds_caption_2} ({M} columns, {C} calculated fields extracted from TWB)
   
   The TWB already contains all column definitions and calculated fields.
   The Tableau API would additionally resolve the physical table structure
   (table names, joins, db/schema paths) — but this is optional.
   
   Do you have access to the Tableau Server hosting these datasources?
     Y  Yes — query the Tableau API for table structure   (requires /ts-profile-tableau)
     N  No  — proceed with TWB metadata only              (common for consultant/remote migrations)
   
   Enter Y / N:
   

2. If the user chooses N (no API access):

   • Log: "Proceeding with TWB-embedded metadata — {M} columns, {C} calculated fields already extracted. Physical table names and joins will be confirmed manually in Steps 3.6 and 4."
   • The TWB's <column> elements (captions, datatypes, roles) and <metadata-records> (local-name → remote-name mapping) provide enough for formula translation and column mapping. The physical table names come from <metadata-record> parent-name attributes (e.g. [Custom SQL Query3], [Table_Name]).
   • Skip to Step 3.6 (join confirmation) — the user will provide or confirm joins and table mappings manually. This is the normal path for consultant/remote migrations.
   • Continue to Step 4 with the TWB column info.

3. For each sqlproxy datasource, extract dbname from the <connection> element, then:

   Progress label: "Querying Tableau API (not ThoughtSpot) to resolve published datasource columns…" — make it clear this is a Tableau Server query, not a ThoughtSpot metadata search.

   # Find the published datasource by name
   ts tableau datasources --profile {PROFILE} --name "{dbname}"
   

   Parse the JSON output to get the datasource id.

   # Get field metadata
   ts tableau datasource {id} --profile {PROFILE} --fields
   

   The fields array contains:

   Field	Use
   fieldCaption	Column display name → ThoughtSpot column name
   dataType	real/integer/string/date/datetime/boolean → TS data type
   columnClass	COLUMN (physical), CALCULATION (formula), BIN, GROUP
   formula	For calculated fields — the Tableau formula text for Step 5 translation

4. Merge the resolved fields into the parsed datasource structure, replacing opaque sqlproxy column references with real names and types. Proceed to Step 4.

5. For textscan (CSV) or excel-direct sources: offer to download the source data for warehouse provisioning. This is essential when the data only exists in Tableau Cloud and has not been loaded into a warehouse:

   # Download the published datasource content
   ts tableau download {datasource_id} --profile {PROFILE} --output-dir {output_dir}
   

   The command downloads the TDSX archive, extracts it, and validates CSV files for row integrity (column count consistency, corrupt lines). Check the validation result:

   • If is_valid: false — report the corrupt lines and offer to auto-fix (strip them) before proceeding to warehouse load. The DunderMifflin live test (2026-06-26) found a corrupt line (1tou) in a Tableau Cloud textscan download — this is a known Tableau artifact.
   • If is_valid: true — proceed; the CSV is clean for loading.

   Surface: "The data for datasource '{name}' is a {type} file hosted on Tableau Cloud. It needs to be loaded into a warehouse table before ThoughtSpot can connect to it. I've downloaded and validated it — {row_count} rows, {status}. Shall I help set up the warehouse table? (This will require a Snowflake/Databricks connection.)"

   If the user says yes, this is the handoff point for BL-010 (ts-load-source-data) when that skill is built. Until then, guide the user through manual warehouse provisioning (CREATE TABLE + stage + COPY INTO for Snowflake, or INSERT VALUES for Databricks).

Prerequisites

• Tableau profile configured via /ts-profile-tableau (optional — skill degrades gracefully)
• ts CLI v0.29.0+ (includes ts tableau build-model with --max-retries, enriched error reporting, GENERATE mode phased output, and --table-name-map)

---

Step 3.6 — Join Confirmation

Joins define the model's query behavior — never silently add joins that aren't in the TWB.

When the TWB parse found joins (from <relation type="join"> or <object-graph><relationships>)

Present them to the user for confirmation:

Joins found in workbook ({N} total):
  1. TABLE_A LEFT JOIN TABLE_B ON TABLE_A.COL = TABLE_B.COL
  2. TABLE_A LEFT JOIN TABLE_C ON TABLE_A.COL = TABLE_C.COL

Do these look correct? (Y/N/Edit)

If the user edits, accept updated join definitions and continue.

When the TWB parse found NO joins (common with published datasources/sqlproxy)

⚠ No join definitions found in the workbook file.
This is normal for published datasources — joins are defined server-side.

Tables in this datasource: TABLE_A, TABLE_B, TABLE_C, ...

To build the model, I need join definitions. Options:
  D  Define joins — I'll suggest based on matching column names, you confirm
  S  Skip joins — create separate single-table models (no multi-table queries)
  P  Provide — paste or describe join definitions

If the user picks D, suggest joins based on shared column names between tables:

Suggested joins (based on shared column names):
  TABLE_A.PROMOTION_ID = TABLE_B.PROMOTION_ID (LEFT_OUTER, MANY_TO_ONE)
  TABLE_A.PROMOTION_ID = TABLE_C.PROMOTION_ID (LEFT_OUTER, MANY_TO_ONE)

Accept suggested joins? (Y/N/Edit)

Range join alternative

When the parse detects date-range filter formulas (e.g., [DATE] >= [START_DATE] AND [DATE] <= [END_DATE]) AND separate fact/dimension tables with start/end date columns, surface an additional option:

💡 Detected date-range filter pattern. ThoughtSpot supports range joins:
  FACT.DATE >= DIM.START_DATE and FACT.DATE < DIM.END_DATE

This is more efficient than a filter formula. Use a range join instead? (Y/N)

See tableau-tml-rules.md "Range join alternative" for the TML syntax.

---

Step 4 — Confirm Source Tables (ask before searching)

This is the first thing after the parse — before selecting a connection, searching ThoughtSpot, or fetching any schema. Getting the order wrong wastes the user's time: scanning the whole instance, or pulling a connection's schema, when the user already knows whether the tables exist is pure overhead (and the connection-schema fetch is slow and can 404). Ask first; act second. This step only asks and confirms — it never loads or modifies warehouse data. It mirrors ts-convert-from-databricks-mv Step 7.

Do NOT run ts metadata search, ts connections list, or ts connections get until the user has answered the question in 4a. No exploratory "let me just check" searches — the answer decides whether any search runs, and at what scope. An ungated ts metadata search --all on a large instance is exactly the wasted work this step exists to prevent.

4a — Present the table inventory and ask

Scope 4 (Models only): tables already exist by definition. Skip choices N and ? — present only E and G and default to G if the user has the GUIDs. The user chose scope 4 specifically because the tables are in ThoughtSpot already.

Show the full inventory of physical tables from Step 3, then ask whether they already exist as ThoughtSpot Table objects:

Source tables referenced by {workbook_name} ({N} total):
  1. P1-UK-Bank-Customers
  …

Do these already exist as ThoughtSpot Table objects?
  E  Exist       — reuse them (I'll search for their GUIDs)
  G  Have GUIDs  — provide GUIDs directly (fastest, no search needed)
  N  Don't exist — create new on a connection            (default)
  ?  Not sure    — search ThoughtSpot to check (avoids creating duplicates)

Enter E / G / N / ? :

If the tables differ in status (some exist, some don't), accept a per-table answer.

4b — Act on the answer

• N (don't exist) → no search. Go to Step 4.5 to pick the connection, then create Table TMLs in Step 5a (the default path).

  Deduplication note: if you are migrating multiple workbooks that share the same published datasource, the tables from the first migration already exist. Choosing N again creates duplicates. If this is a second (or later) workbook migration, consider E, G, or ? instead to reuse the tables already in ThoughtSpot.

• G (have GUIDs) → no search. For each table, ask the user to provide the GUID (the id value from ThoughtSpot, e.g. from a previous migration or from the UI). Use the provided GUIDs in the model's model_tables[] entries and skip generating Table TMLs for those tables. If the user has GUIDs for some tables but not all, treat the remaining tables as N (create via Step 4.5 + 5a).

• E (exist) → search to find the GUIDs — but choose the scope first (4c). Searching ensures the tables are not duplicated and resolves their GUIDs automatically.

• ? (not sure) → search — choose the scope first (4c). Report what was / wasn't found; reuse the found ones, treat not-found tables as create (Step 4.5 + 5a). This is the safest option when migrating into an instance that may already have some of these tables.

4c — Choose the search scope (E and ? paths only)

A whole-instance scan is the slow path. Always offer the narrower option, and search by table-name pattern (--name) so the API does the filtering — never pull every table and filter locally:

How should I search for these tables?
  C  Within a specific connection — fastest; I'll list connections and search that one
  I  Entire ThoughtSpot instance  — broader, slower

Enter C / I :

# Targeted by name — both scopes start here (NOT `--all`)
source ~/.zshenv && ts metadata search --subtype ONE_TO_ONE_LOGICAL --name "%{table_name}%" --profile {profile_name}

• C (within a connection) → first identify the connection using the N (name it) / F (filter by substring) / L (list all) prompt in Step 4.5 — present that prompt and let the user choose; do NOT run ts connections list and dump every connection by default. Once the connection is chosen, run the name search above and keep only results whose metadata_header.dataSourceName equals the chosen connection name (verified 2026-06-16: each search result carries its connection in metadata_header.dataSourceName, e.g. "APJ_TAB"). Fastest, and unambiguous when the same table name exists on several connections.
• I (entire instance) → run the name search above with no connection filter.

Match on table name (metadata_name) and, for the connection scope, metadata_header.dataSourceName. (db/schema also appear in metadata_header — database_stripes / schema_stripes — use them to disambiguate same-named tables within one connection.) For each table found, reuse its name/GUID in the model's model_tables[] and skip generating a Table TML for it in Step 5a. If a table the user said Exists is not found, say so and confirm before falling back to create.

Only fall back to --all (fetch every table) when no usable name pattern can be formed (e.g. the name is too generic). Tell the user that cost before running it.

4d — Confirm any missing sources before proceeding

If any table the plan intends to create is not found on the chosen connection, surface it and require confirmation — this is the silent-failure case (a model TML that points at a table the connection can't see still imports cleanly, yet every search and liveboard built on it comes back empty):

⚠ The following table(s) are not visible to connection "{connection_name}":
    - {db}.{schema}.{db_table}
  Their models will import, but searches return no data until the data is loaded
  and visible to the connection. This skill does not load data.

  Proceed anyway (generate the TMLs as-is)?   (yes / no):

Do not proceed past this warning without the user's confirmation.

---

Step 4.5 — Select ThoughtSpot Connection (create path or connection-scoped search)

Run this only when a table will be created (the N path, or tables not found on the E / ? paths) or to scope a connection search in 4c. If every table was matched to an existing object, skip this step — reusing tables needs no connection work.

First: use an existing connection or create a new one. Ask:

The generated tables need a ThoughtSpot connection that can reach the source database.
  E  Use an existing connection
  C  Create a new connection   (Snowflake source only, key-pair auth)

Enter E / C:

When to create: a ThoughtSpot connection only sees databases its warehouse role is granted. If no existing connection's role can see the source database, table creation fails with "Database … does not exist in connection" — that is the signal to create one (do not trial-and-error existing connections to find out).

C — create a new connection. Supported here for Snowflake sources via key-pair auth. Collect the connection name, Snowflake account identifier, user, role, warehouse, and the path to the unencrypted PKCS#8 private key (.p8), then run:

source ~/.zshenv && ts connections create \
  --name "{connection_name}" \
  --account "{account}" --user "{user}" --role "{role}" --warehouse "{warehouse}" \
  --database "{database}" \
  --private-key-path "{key_path}" \
  --profile {profile_name}

The role must have USAGE on the database/schema (and SELECT on the tables). The matching public key must already be registered on the Snowflake user. Credential handling (required): never ask the user to paste a private key, password, or secret into the conversation — the key is passed by file path only and ts connections create never echoes it. If the source is not Snowflake, or password/OAuth is required, connection creation is out of this skill's scope: direct the user to create the connection in the ThoughtSpot UI, then return on the E path. Use the returned name as {connection_name}.

E — use an existing connection. Don't dump the full list by default — a long connection list is noise when the user already knows the one they want. Ask:

How would you like to identify the connection?
  N  Name it     — type the exact connection name; I'll use it directly
  F  Filter      — give a partial string; I'll list only connections that match
  L  List all    — show every connection and pick by number
  T  Trust       — type the name and skip validation (faster — import will fail
                   cleanly if the name is wrong)

Enter N / F / L / T:

T — trust the name. Use the typed name directly without running ts connections list. This skips validation but is faster on instances with many connections. The import will return a clear error ("Connection 'X' not found") if the name is wrong.

Compound prompt (N or T path). When the user takes the N or T path, offer the db/schema confirmation in the same prompt to eliminate sequential questions:

Connection: ____________  (exact ThoughtSpot connection name)
Database:   ____________  (or press Enter to use '{twb_extracted_db}')
Schema:     ____________  (or press Enter to use '{twb_extracted_schema}')

This replaces the separate db/schema confirmation loop below when the user provides all three in one response.

For N/F/L, fetch the connections once (auto-paginated, returns all):

source ~/.zshenv && ts connections list --profile {profile_name}

Resolve the user's choice against that result:

• N (name it) — match the typed name against the returned name values (case-sensitive). Exactly one match → use it. No match → show the closest names and re-ask. Don't fabricate a name the list doesn't contain — the table TML needs the exact, case-sensitive connection name.

• F (filter) — keep connections whose name contains the string (case-insensitive), show them as a short numbered list (name, type, database), and pick from that. One match → auto-select and confirm; none → widen the string or switch to L.

• L (list all) — show the full numbered list and pick by number:

  Available ThoughtSpot connections:
    1. SNOWFLAKE_PROD    (RDBMS_SNOWFLAKE)   — PROD_DB
    2. ANALYTICS_DW      (RDBMS_SNOWFLAKE)   — ANALYTICS_DB
  
  Which connection should the generated tables use? (Enter number):
  

If only one connection exists in total, auto-select it and confirm regardless of the choice. Save the selected connection's exact name value as {connection_name}.

Resolving db / schema / table for new tables. Each new table needs the {db}, {schema}, and {db_table} it maps to on the chosen connection. The Tableau workbook contains the source environment's database paths — these may not match the target ThoughtSpot connection (e.g. a consultant running the migration in their own environment with a different database). Always confirm before using them.

Show the TWB-extracted paths and ask:

The Tableau workbook references these source database paths:
  - {source_db}.{source_schema}.{table_1}
  - {source_db}.{source_schema}.{table_2}
  …

Do these match your ThoughtSpot connection's database and schema?
  Y  Yes — use these paths as-is
  D  Different database/schema — I'll provide the correct values
  T  Per-table — some match, some don't (I'll confirm each)

Enter Y / D / T:

• Y → use the TWB-extracted {db}, {schema}, and {db_table} values directly.
• D → ask for the target {db} and {schema} once. Apply them to all tables (table names stay the same unless the user overrides). This is the common consultant scenario where all tables live in the same database but under a different name.
• T → walk through each table and confirm or override its {db}, {schema}, and {db_table}. Use this when tables span multiple databases or schemas in the target.

If the user doesn't know the correct paths:

1. Ask the user for the db / schema (and table name if it differs from the source) — usually instant, and they know it.
2. Only if they're unsure, fetch the connection schema to resolve names:

   source ~/.zshenv && ts connections get {connection_id} --profile {profile_name}
   

   This uses the v1 fetchConnection endpoint — it can be slow and returns 404 on some connection types, so treat it as the fallback, not the default. If it returns no tables (empty externalDatabases) or fails, ask the user for the names directly.

A connection is required for any table being created — there is no skip path. ThoughtSpot tables are logical objects over a live connection: the physical table must already exist in the database and the connection must already exist for the table to be created at all. Do not offer placeholders or a dry-run mode — they only produce objects that can never bind to data. If the user has no suitable connection: for a Snowflake source, create one via the C path above (key-pair auth); for any other source, or when password/OAuth is required, stop and tell them the connection must be created first in the ThoughtSpot UI (that connection setup is out of this skill's scope).

Use the connection's exact name in every table TML and SQL View TML — never a GUID. The v2 API cannot search connections by name, so the name string is both necessary and sufficient; do not try to resolve it to an ID. See ../../shared/schemas/thoughtspot-table-tml.md "Connection Reference".

---

Step 5 — Generate TML Files

Create output directory:

mkdir -p /tmp/ts_tableau_mig/output/{workbook_name}

5a. Table TML — one per physical table (skip custom SQL relations)

Scope gate: runs for scopes 1, 2, 5. Skip for scope 3 (LB only — no tables) and scope 4 (Models only — tables already exist; use GUIDs from Step 4).

For each physical table identified in Step 3 with type="table", generate a .table.tml file. Skip custom SQL relations — those are handled in Step 5c.

Follow all rules in tableau-tml-rules.md.

Template:

table:
  name: TABLE_NAME
  db: RESOLVED_DATABASE
  schema: RESOLVED_SCHEMA
  db_table: physical_table_name
  connection:
    name: "{connection_name}"       # exact ThoughtSpot connection name, case-sensitive — NOT a GUID
  columns:
  - name: COLUMN_NAME
    db_column_name: COLUMN_NAME
    data_type: VARCHAR              # VARCHAR | INT64 | DOUBLE | FLOAT | BOOL | DATE | DATETIME
    properties:
      column_type: ATTRIBUTE        # or MEASURE
      aggregation: SUM              # only if MEASURE — SUM | AVERAGE | COUNT
    db_column_properties:
      data_type: VARCHAR            # must match data_type above

Key rules:

• connection.name is required — a ThoughtSpot logical table must sit on a connection that already exposes the physical table and its columns. Use the connection name directly (case-sensitive); never look up a GUID — the v2 API cannot search connections by name, and the name is what the TML needs. See ../../shared/schemas/thoughtspot-table-tml.md "Connection Reference".
• Use the db, schema values resolved from Step 4.5 (the connection is required, so these are always real).
• db_column_name must match the physical column the connection exposes — not the Tableau name. When a file source (CSV/Excel) was loaded into the warehouse, the loader usually normalizes names (Item Type → ITEM_TYPE: spaces→_, upper-cased). Use the warehouse column name for db_column_name (and the friendly Tableau caption for the model column's display name). If unsure, the connection schema from Step 4.5 (externalDatabases) lists the real column names; validation reports column not found in connection when they don't match.
• Date stored as VARCHAR — flag it. If the Tableau column is typed date/datetime but the warehouse column is VARCHAR (common when a CSV date loaded as text), binding it as VARCHAR loses all date capability (no buckets/trends/relative-date filters; Spotter won't read it as time). The TS column data_type must match the physical column, so you can't just declare DATE. Surface it and offer: (a) retype at the source (warehouse ALTER/reload to a real DATE — outside this skill; needs the user) then bind as DATE, or (b) keep VARCHAR and add a to_date([col]) derived formula column for date analytics. Don't silently bind a date as a string.
• Partial date strings must produce a full YYYY-MM-DD date. When a source column contains a year-only value (e.g. _2016_17, FY2016, 2016) and needs to become a DATE, always append -01-01 (or -01 for year-month) to produce a complete date. A bare-year conversion like to_date('2016', 'yyyy') produces an ambiguous value that ThoughtSpot cannot bucket (.yearly, .monthly), use for KPI sparklines, or filter as a date range. If the datasource already uses a SQL View, apply the conversion in the SQL query (TO_DATE(SUBSTRING(col, 2, 4) || '-01-01', 'YYYY-MM-DD')). If it uses a regular table, apply it as a model formula (to_date ( concat ( substr ( [col] , 1 , 4 ) , '-01-01' ) , 'yyyy-MM-dd' )). See tableau-tml-rules.md "Date Column Rules" for the full pattern table.
• Use INT64 for Tableau integer — never INT