| name | project-data |
| description | The one skill for a Spryker project's data/import/** — use whenever demo/import data must be populated, reshaped, reduced, cleaned up, or removed. Triggers: 'adapt the demo shop to my stores and currencies', 'add demo data / generate a catalog of tools/books/dresses from these images', 'create CMS blocks from these pictures', 'start clean with no demo catalog', 'we only sell heat-recovery, drop the rest of the catalog', 'remove the demo customers / merchants / reviews', or any ad-hoc edit of import CSVs. Used by the wizard's data step and standalone on a fresh or running project. |
project-data
The universal skill for everything under the project's data/import/**. It owns one job — get the project's import data into the shape the project needs — and it does that through one of five strategies. You pick the strategy from the interview (.ai-dev/project-setup.md data.mode) or from what the user asks in plain terms, apply the shared core below (tools, discipline, state-reading, apply-ladder, invariants — the same for every strategy), then follow that strategy's reference file for the method.
You drive the spryker-import-tools scripts (csv.php + validate.php) with judgment; the tools are concept-free (they know nothing about stores/locales/currencies) — the judgment is what this skill supplies. (Failure-signature triage for the data steps lives in the Known-traps catalog: ../project-starter-wizard/references/pitfalls.md.)
Pick the strategy (interview or plain-language intent → strategy)
| The user wants to… | Strategy | Method reference |
|---|
| Reshape the shipped demo catalog to the project's stores/locales/currencies (English content everywhere) | adapt | references/adapt.md (+ references/adapt-strategy.md) |
| Author a fresh themed catalog in the project's own vertical — "add demo data", "generate ~20 products from these images", "create CMS blocks from these pictures" | generate ⚠ (experimental — supervised only; see reference) | references/generate.md |
| Start with no demo catalog — a minimal shop that still boots green (working email/tax/payment/shipment, project stores, empty catalog) | clean | references/clean.md (spec: ../define-stores/references/minimal-baseline.md) |
| Keep only part of the demo catalog and remove the rest ("we sell X, drop the other categories") | reduce | references/reduce.md |
| Remove whole demo domains — customers, merchants, reviews, wishlists, CMS, discounts, transactional activity, … | cleanup | references/cleanup.md |
| Remove a locale/language — "drop German", "we don't need de_DE" | adapt's strip pass, standalone | references/adapt.md step 5 (columns via drop-columns --suffix, locale-ROW files, glossary rows, hybrids like product_image whose rows define image sets) |
Ad-hoc add / edit / filter / one-off remove on import data that isn't one of the above is the same tools driven directly (csv filter/delete/set/duplicate-* → validate → apply) — no separate strategy; use the shared core and the closest reference for the cascade rules.
- Standalone (a user asks directly), classify their words against the table — never make them name a strategy or a file. "Add demo data for my shop" → generate. "Drop the demo customers" → cleanup. "We only sell heat-recovery" → reduce. If genuinely ambiguous, ask one plain question.
- In the wizard,
data.mode picks it: adapt→adapt, clean→clean, generate→generate, leave→skip entirely; plus a reduce pass after adapt if reduce_catalog ≠ keep:all.
Strategies compose — not a one-time exclusive choice
These are operations on the current data, runnable in any order, whenever: adapt now and later replace with generate; clean the catalog and then generate onto the cleared base; reduce after adapting. So do not assume a starting state — the same request means different work depending on what's already there. Every strategy therefore begins by reading the current state (below). The one still-unbuilt path is merging a fresh themed catalog alongside a still-full adapted catalog (both live at once, sharing namespaces) — clean or reduce first, then generate.
Shared core (applies to every strategy)
Tools & command discipline
Follow spryker-import-tools → "Invocation & command discipline" (the authoritative copy). In short: invoke csv.php/validate.php by their literal path from the project cwd ($CSV/$VALIDATE in the references = that path, substituted inline — never a shell variable, never cd); one op over many files = one command with --in-place, never a shell loop or operator; count via rowCount/matchedRows; explore with the built-in Read/Grep/Glob; YAML via the built-in Edit tool; every rm surfaced as an explicit deletion step. Work from real files: discover columns with csv columns, never hardcode a header. One rule specific to this skill:
- Do NOT spelunk git history — work from the current files (
csv read/Read), never git show HEAD:<file> (and never git show … | head). It's a fresh git clone, so git is the undo for FILE edits — never hand-roll a cp-into-scratchpad backup. But git does not restore a database or a search volume — a reset/clean-data wipe is NOT git-recoverable (see the destructive-op gate below).
When a change breaks something, suspect your own data delta first
If the storefront worked in the fresh demoshop and breaks after your import edits (empty menu, missing homepage blocks, unbuyable product), the file you just changed/dropped is the prime suspect — not vendor/core code. Diff your delta (git diff -- data/import config) and re-check completeness (preflight, refs, per-store×locale counts) before reading any vendor package. "It worked before my changes" is ground truth (full diagnostic order: boot-and-verify intro). A dropped navigation import set, for instance, presents as a broken menu that looks like a vendor bug but is your delta.
Record progress for resume (wizard runs)
The data strategies are the longest steps of a wizard run. When running under the wizard, set the state-file step to in-progress and update its note after each major pass (e.g. locales done, currencies pending) — an interrupted run then resumes from the note instead of blindly re-running, which matters most for the deletion/strip passes (not idempotent).
Read the current state first
Before deciding what to do, determine what's actually in data/import/** now — csv count <live catalog files> --plain on the active import dirs and/or the booted search-doc count. Never trust .ai-dev/project-setup.md for the current catalog state — it's a snapshot that a later cleanup/adapt may have invalidated (it can still show a stale "full demo" count long after a cleanup stripped the catalog). What each strategy means depends on the live base (e.g. generate onto a cleaned base = plain add; against a full demo = replace).
Destructive operations — explain, then ask (every time)
Apply the destructive-op gate (authoritative copy: spryker-import-tools → Invocation & command discipline; reset/clean-data weight + escape hatch: boot-and-verify §3b): before any in-place data removal/truncation or any DB/volume drop, preview it (csv filter/delete with no --out/--in-place reports matchedRows without writing), tell the developer in ONE plain line what will be destroyed, and get an explicit go-ahead — even if the allowlist would let it through silently. Never run a destructive step unannounced.
Apply — pre-boot OR post-boot (the reset ladder)
Every strategy works both pre-boot (un-booted clone, e.g. the wizard — the files ARE the state; the first boot imports them) and post-boot (already-booted project — edit files, then reflect). Post-boot follow the reset ladder — full detail in boot-and-verify §3b:
docker/sdk console data:import -c <config> — validates the edits leave no orphan (fast, collects all failures, no teardown).- For deletions, a plain
data:import only upserts and won't remove already-imported rows → reflect them with docker/sdk reset (drops the DB and re-imports). reset needs a TTY: run script -q .ai-dev/reset.log docker/sdk reset in the background, never a plain background shell (else failed to get console); and confirm what reset does in this SDK first — in some it's a full destructive teardown (wipes volumes).
- Drain the queue workers so KV/search catch up. Imports emit publish events automatically —
publish:trigger-events is recovery-only (out-of-sync read model / after a search-index rebuild), never routine.
Never clean-data + full up for a data-only change (~30–60 min rebuild) — reserve that for code/deploy/service changes.
Cross-cutting invariants (they bite in more than one strategy)
Don't check these by hand file-by-file — run php "$VALIDATE" preflight <import-config.yml> and it sweeps the first four across every file the manifest imports in one call (url-uniqueness, is_searchable, price completeness, import order), returning one grouped verdict. The list below is what it enforces and what to drill into per-file when it flags something:
spy_url is globally unique across ALL entities — one duplicate url.<locale> aborts the 30–60 min install. Run validate unique on every url-family file (scan for them; the missed merchant.csv is a classic boot-blocker), not a fixed list.is_searchable.<locale> is load-bearing — blank imports as silently unsearchable; must be copied, never left empty (validate required).- A blank per-locale cell is a FINDING until proven to be a legitimate inherit. The suite already knows this twice and has never generalized it: blank
is_searchable.<locale> imports as silently unsearchable (above), and product-concrete's localized-attribute triplet (name/description/is_searchable) is all-or-nothing per locale — one blank drops the whole locale bucket (references/generate.md C9; content-banner's locale buckets behave the same). So for every .<locale> column set: an empty cell is either a proven inherit (name the importer that inherits it) or a defect. Check with validate required <file> --column <col>.<locale> per project locale; for price columns treat a literal 0 as blank too.
- Account for EVERY row of a structural file you REWRITE (as opposed to add to) — keep, retarget, or consciously drop with a reason, and report the counts. Applies to
navigation_node, cms_slot_block, content_navigation, navigation, cms_block, cms_page, and every other file a strategy rewrites rather than appends to. The failure shape is always the same: the run rewrites the rows the brief named and silently drops the rest — a brief to "replace the 3 category links" wipes the footer's Imprint/GTC/Privacy links, social icons and payment-partner logos — and a green boot never notices, because the surviving rows import fine. Classify by the file's own type column where it has one (navigation_node.node_type: category/cms_page are retargetable, external_url/link are structural keeps carried verbatim). Detectable: in-count must equal , in writing.
Consolidate & clean up (the tail of the data work — this pass SUPERSEDES adapt's in-place rule)
Sequencing note: adapt's "do not create a data/import/<project>/ tree" governs during its in-place transform (git is the diff/revert there); this consolidation runs after the strategy completes, and moving to one project tree is then correct — two phases, not a contradiction. What the pass fixes: the imported sources end up scattered across several trees (common/<REGION>/, common/<STORE>/, common/common/, a themed <project>/catalog/), and define-stores' cp -r single-source seeding copies the WHOLE canonical dir per store — so each seeded store dir carries stale demo files the manifest never imports. Left alone this is a second, plausible-looking source of truth — most files in a seeded store dir can be stale demo copies (a large product_price.csv, demo SKUs, demo merchants/offer-prices) that mask other problems. So finish the data work by consolidating:
This pass runs at the end of every strategy (adapt, generate, clean, reduce). Its purpose is a tree a developer can read: without it, live generated data sits under data/import/<project>/ while reused-but-retargeted structural files stay in common/** and emptied or never-imported demo files remain scattered, so which data is actually imported is unclear.
The active manifest (full_<REGION>.yml) is the single source of truth; after this pass the on-disk tree matches it. Any file under the import roots that no active manifest references is removed — unless it's a deliberately-kept CI/fixture bucket (b2b_common/, robot/), which is noted in the summary.
- One tree. Every
source: the active manifest imports lives under one project tree: data/import/<project>/{stores/{REGION,STORE…},shared,catalog,cms,stripe}/. This includes the reused structural files: generate keeps them in place while authoring, and this pass moves the retargeted ones into the project tree and repoints the manifest, leaving common/** with nothing the run depends on.
- Delete stale unreferenced files.
define-stores' cp -r single-source seeding copies the whole canonical dir per store, so most of each seeded store dir is demo files the manifest never imports; remove those and any leftover demo dirs. Prefer deleting a file (and its source:) over emptying it to a header, unless the importer requires the entity present. Surface each rm/rm -rf per the destructive-op gate.
- Verify the tree matches the manifest with
validate.php orphan-files <import-config.yml> <root> — it lists every CSV under the root that the manifest doesn't import. A clean run (0 findings) proves the on-disk tree equals the active manifest; each finding is a dead file to delete or a source to add. Report the final count (e.g. "N files, all imported by full_<REGION>.yml").
Not here (kept as separate skills)
Store definitions / region / deploy & import config skeleton — define-stores (its references/minimal-baseline.md is the clean-strategy spec). Real translation of a locale's content — translate-content (opt-in, after boot). Go-live production-data curation (real tax, licensed imagery, demo-password reset) — curate-golive-data. The engines this skill drives — spryker-import-tools (csv.php + validate.php); booting and per-store verification — boot-and-verify.
