manage-shops

name

manage-shops

description

How to add a new shop to this shopping project's data store, or remove an existing one. Use this whenever the user wants to onboard / add / import a new shop or store, remove / delete one, or asks how to handle a shop that isn't Shopify (a small, bespoke, or hand-built website). Covers detecting and ingesting a Shopify shop, a decision ladder for non-Shopify shops, the db.upsert_product / db.ingest_order extension contract, and remove_store.py. Reach for this even if the user just says something like "I started buying from <some shop>, can we add it?" or "get rid of the Acme Foods data". Not for shops already in the store — querying purchases, comparing prices, exporting, counting shops, refreshing a catalogue, or fixing an existing parser are different tasks.

Adding and removing shops

This project keeps every shop's orders and product catalogue in one SQLite store (shopping.db), with shop as data (a shop_id column), not schema. So onboarding a shop never needs new tables — it needs new rows.

The one principle: everything funnels through two functions

However you obtain a shop's data — a tidy API, scraping, or typing it in by hand — your job is finished when you've called these, defined in db.py:

• Catalogue (products you could buy): db.upsert_product(...)
• Purchase history (what the user actually bought): db.ingest_order(...)

Get the data into the shape these expect and the rest (joins, price-change queries, delisting, coverage reports) just works. Don't invent a parallel path.

import db
conn = db.connect()                      # opens shopping.db
db.init_schema(conn)                     # safe to call every time
shop_id = db.get_or_create_shop(conn, slug="acme", name="Acme Foods",
                                 platform="shopify", base_url="https://acme.example")

# A catalogue product (variants is a list; each needs at least a sku):
db.upsert_product(
    conn, shop_id, base_url="https://acme.example",
    handle="acme-chilli-oil", title="Acme Chilli Oil, 250ml",
    description="Plain-text description.", vendor="Acme", product_type="Condiments",
    raw={...},                            # the full source record, kept in raw_json
    fetched_at="2026-06-08T12:00:00",
    variants=[{"sku": "ACM001", "price": 6.99, "compare_at_price": None, "available": True}],
)

# An order: build Order/Item (reuse the dataclasses in extract_souschef.py) then:
db.ingest_order(conn, shop_id, order)    # order.items is a list of Item

Both are idempotent — re-running replaces an order's items and upserts products, so you can re-ingest freely.

Catalogue vs purchase history — they generalise very differently

This distinction trips people up, so internalise it:

• The catalogue side generalises well. Every Shopify shop exposes the same /products.json, so one fetcher (catalogue.py) serves them all.
• Purchase history does NOT generalise — even between two Shopify shops. Whether you can get order history depends on the shop's theme and settings, not the platform. One Shopify shop might offer downloadable PDF invoices while another, also on Shopify, offers nothing to download at all. So "how do I get the orders?" is always a per-shop question, and many shops simply won't have an automatable answer.

That's why a shop's pdf_dir is optional and catalogue-only shops are first-class. A shop with a catalogue but no purchase history is completely fine — you just won't have personal buying data to enrich.

Adding a Shopify shop (the easy path)

Most foodie shops are Shopify. Confirm it, then it's a one-line registry entry.

1. Detect Shopify and check you're allowed. Verify rather than assume:

   curl -s 'https://SHOP/products.json?limit=1'        # JSON with "products" => Shopify
   curl -s 'https://SHOP' | grep -o -i -m1 'Shopify\.shop[^;]*'   # backup tell
   curl -s 'https://SHOP/robots.txt'                   # confirm /products.json isn't Disallowed for *
   

   /products.json is normally allowed; the usual Disallows are /cart, /checkout, /search etc. If it's genuinely blocked, treat it as a non-Shopify shop (below) and respect that.

2. Sanity-check the size before committing to a full pull, so you know what you're asking of a small producer's site:

   curl -s 'https://SHOP/products.json?limit=250&page=1' | python3 -c 'import sys,json; print(len(json.load(sys.stdin)["products"]),"on page 1")'
   

   A full pull is JSON only (no images), paced at 1 req/sec, and cached to data/<slug>_catalogue.json so repeat runs never re-hit the shop. Even a few thousand products is a handful of MB — gentler than one human browsing.

3. Register it — add one entry to the SHOPS list in ingest.py:

   dict(slug="acme", name="Acme Foods", platform="shopify",
        base_url="https://acme.example", pdf_dir=None),   # pdf_dir set later if PDFs exist
   

4. Ingest and verify:

   uv run python ingest.py --shop acme            # add --refresh to force a re-fetch
   

   Check the per-shop totals and (if it has purchases) the coverage line it prints. Spot-check with a quick query that a product has a title, url and price.

Adding a non-Shopify shop (the "12-year-old niece built it" path)

No /products.json? Work down this ladder — stop at the first rung that yields clean data. The goal is always the same: produce records and call db.upsert_product.

1. Is it secretly a known platform? Many bespoke-looking sites run on a platform with a machine-readable feed. Probe for the tells:

   • WooCommerce / WordPress: curl -s 'https://SHOP/wp-json/wc/store/products?per_page=1' (the public Store API often returns JSON), or look for wp-content / wp-json.
   • Squarespace: append ?format=json to a shop or product URL.
   • Wix: parastorage.com / _api in the page source.
   • Ecwid / BigCommerce / Etsy: each has its own store API or embedded product JSON — search the page source for a likely tell, then verify.

2. No platform API, but structured data in the pages? Check, in order: JSON-LD (<script type="application/ld+json"> — Shopify-style Product objects, the cleanest), Open Graph <meta> tags, microdata, a sitemap.xml listing product URLs, or a downloadable price list / CSV. (A single Shopify product page will often hand over its description this way before you find the bulk feed — see catalogue.html_to_text for the HTML→text helper.)

3. Genuinely bespoke static HTML with nothing structured? Then it's almost certainly small — that's the whole reason it was hand-built. Don't write a fragile scraper for three soaps. Either save the few product pages and parse them with stdlib (html.parser, re), or just hand-enter them: a tiny seed script that calls db.upsert_product for each product, or a small CSV the script reads. Five minutes of typing beats an afternoon of brittle XPath.

Whichever rung you land on, write a small per-shop fetch/parse step (mirroring catalogue.py) that emits the product shape and calls db.upsert_product. Wire it into ingest.py behind the shop's platform value so --shop <slug> drives it like any other.

Purchase history for any shop

Orders are per-shop by nature (see above). Keep a shop's invoice PDFs in orders/<slug>/ — that folder is git-ignored, so personal invoices (which carry names, addresses and payment references) never get committed — and point the shop's pdf_dir there. When a shop does give you order data:

• PDF invoices: write a shop-specific parser. The existing one (extract_souschef.py) is tuned to one shop's invoice layout — its column/x-position tricks won't transfer to a different layout. Reuse its Order/Item dataclasses, build them from the new layout, and call db.ingest_order.
• Email receipts / a CSV export / hand-entry: same destination — construct Order/Item objects and call db.ingest_order.

If a shop has no obtainable history (a catalogue-only shop), skip it — leave pdf_dir=None and rely on the catalogue.

Removing a shop

A bare DELETE FROM shops is blocked (orders and products reference it without a cascade), so use the dedicated command. It's non-interactive (no prompt to stall an agent) and dry-run by default:

uv run python remove_store.py acme          # preview: shows what WOULD be deleted
uv run python remove_store.py acme --yes    # actually delete it

--yes removes the shop and all its orders, items, products and variants in one transaction. The data is re-derivable (re-run ingest.py), so this isn't precious — but the dry-run preview is there so you see the blast radius first.

Files you'll touch

• ingest.py — the SHOPS registry and the ingest CLI.
• db.py — the store: upsert_product, ingest_order, get_or_create_shop, mark_delisted, remove_shop.
• catalogue.py — Shopify fetch + html_to_text (template for other platforms).
• extract_souschef.py — the (shop-specific) PDF parser and Order/Item.
• remove_store.py — the removal CLI.