source-and-sum

name

source-and-sum

description

Analyze files of tabular data (CSV, TSV, JSON exports, bank/credit-card statements, logs, metrics exports) using @source-and-sum/core, so every step is named and recorded and every number in the answer traces back to the raw rows that produced it. Use whenever the user asks to analyze, summarize, categorize, reconcile, or find patterns in data files — and especially when they want to check the work afterwards.

source-and-sum

Write the analysis with recorded steps instead of ad-hoc scripting. The user then audits your work by looking at the data each step produced, not by reading your code.

That is the whole point. An analysis that produces the right answer but cannot be checked has failed at the thing this library exists for.

When to use this

Use it when the user wants a real answer out of real data files: spending breakdowns, log summaries, export reconciliation, metric rollups, "which of these am I still paying for".

Don't use it for a one-line grep/wc question, for data small enough to just read, or for non-tabular work. Setting up a pipeline for three rows is noise.

Setup

One command, from the directory that holds the data:

npm create source-and-sum

It writes a package.json if there isn't one, installs the library (and a CSV parser only if it sees CSVs), lists the data files it found, and tells you whether to write analyze.ts or analyze.mjs for the Node in this shell. It does not write the analysis — that is yours.

Doing it by hand, if you prefer:

# Only if there is no package.json in THIS directory. Without one, `npm install`
# walks up the tree and installs somewhere you did not mean.
npm init -y && npm pkg set type=module

npm install @source-and-sum/core
npm install csv-parse            # only if the inputs are CSV

Use npm unless you can see a pnpm-lock.yaml, yarn.lock or bun.lockb in the directory — a data folder usually has no package manager preference, and guessing wrong fails on a machine that has only npm.

Which file extension. node analyze.ts works on Node ≥22.18, which strips types natively. Check with node --version before writing the file; on anything older write analyze.mjs in plain JS, because a .ts file that Node cannot run fails with an error that looks like a problem with the library.

Never modify the user's source files. Read them; write only the artifact.

The artifact contains the user's actual data — statement rows, salaries, whatever they gave you. If the folder is a git repository, add .source-and-sum/ to .gitignore before running, and say that you did.

The shape of a script

import { analysis } from "@source-and-sum/core";
import { parse } from "csv-parse/sync";

const a = analysis("Spending 2026");

const citi = a
  .file("statements/citi.csv")
  .flatMap("parse file", (f) => parse(f.text, { columns: true }) as Row[])
  .filter("Drop payments and credits", (r) => r.Debit !== "")
  .map("Normalize Citi charges", (r) => ({
    month: `${r.Date.slice(6, 10)}-${r.Date.slice(0, 2)}`,
    merchant: merchantOf(r.Description),
    cents: Math.round(Number(r.Debit) * 100),
    raw: r.Debit,               // keep the source value next to the derived one
  }));

citi
  .(,  t.)
  .(,  cents + t., )
  .(,  y..(x.))
  .(,  ({ : r., : r. /  }))
  .(, { : , : , :  });

 a.();

report() prints a step summary and writes .source-and-sum/runs/<runId>/.

If any step can throw, put the pipeline in a try and report() in the finally — a run that dies without an artifact is the one you most needed to look at.

Re-running after a fix keeps both runs. Each report() writes a new timestamped directory; the newest 30 are kept. That is what lets the user ask whether your fix changed the step, so do not pass runId to overwrite the previous run. Do pass a label — it is what the run picker shows, and 2026-07-27T14-12-34 on its own says nothing about what you changed:

await a.report({ label: "amount column parsed as cents" });

Primitives

a.file(path)	one row: { path, name, bytes, text }
a.source(name, rows)	in-memory rows (lookup tables, constants)
.map(name, (row, i) => out)	1:1
.filter(name, (row, i) => bool)	keep rows
.flatMap(name, (row, i) => out[])	1:N — use for parsing
.sort(name, (a, b) => n) / .topK(name, k, cmp)	ordering
.topKBy(name, k, (row) => key, "highest" | "lowest")	top N — prefer this over topK
.distinct(name, (row) => key)	one row per key, keeping the first
.groupBy(name, (row) => key)	→ { key, items }[]
.reduce(name, (acc, row, i) => acc, init)	→ one row
.reduceGroups(name, (acc, item, i) => acc, init)	→ { key, value }[]; init must be a factory (() => ({…})) for a mutable accumulator
.join(name, right, { left, right, select }) / .leftJoin(…)	select gets undefined on the right for unmatched rows in a left join
.union(name, ...others)	concatenate a fixed set; types may differ
a.union(name, collections)	concatenate a list built at runtime
.intersect(name, right, { left, right })	keep this collection's rows whose key is also on the right
.difference(name, right, { left, right })	keep this collection's rows whose key is not on the right
.apply(name, (rows) => out[])	escape hatch — breaks row-level lineage

Steps run eagerly, so collection.rows is real data at any point — log it while you are working.

Use topKBy, not topK. A comparator can be written backwards, and when it is, you get the bottom five under a step you named "Top 5 categories" — real rows, exact lineage, a chart that agrees with them, and nothing anywhere to tell you. .topKBy("Top 5 categories", 5, (r) => r.value, "highest") cannot be inverted by accident. Reach for topK only when the ranking genuinely needs a multi-key comparator. Ranking keys must be a number, string, bigint, boolean or Date, and every row must give the same kind — if a row's key can be missing or NaN, filter those rows in their own step first (you want that count visible anyway).

distinct keeps the first row carrying each key: "unique merchants", "one row per id". Prefer it over groupBy + map when you want the rows rather than the groups, and never reach for apply to deduplicate — distinct keeps exact lineage and apply throws it away. Its lineage names every row with the key, so tracing a merchant row shows you all its charges, and meta.removed tells you how many duplicates there were.

intersect and difference keep this collection's rows; the right-hand side only decides which of them survive, and contributes no values. Reach for them where you would otherwise build an array outside a step and filter against it (rule 4) — "charges at merchants the lookup has never heard of" is a difference, and written that way the lookup is a visible input with a row count, so a lookup that arrived empty is on screen instead of silently removing nothing. Neither one deduplicates: two rows with the same key both survive.

Marks

A chart names columns and what they mean. It never sets a scale, a domain, a format, or a colour — those are derived from the rows, which is what keeps the picture and the table under it from disagreeing.

Mark	What it is for	Channels
bar	ranking one measure across categories	x category, y magnitude
line	a measure over time	x position, y magnitude, color series
point	one measure against another	x, y magnitudes, color identity
cell	two categories indexing one number (a heatmap)	x, y categories, color magnitude
table	the rows alone	—

A channel is a column name, or { field, type, title } when you need to correct the inferred type ("quantitative" | "nominal" | "temporal") or label an axis:

.chart("Spending by month and category", {
  type: "cell",
  x: { field: "month", type: "temporal" },
  y: "category",
  color: { field: "dollars", title: "spent" },
});

Three things the viewer will do that are worth knowing before you are surprised by them. bar ignores color — a set of nominal bars is one series; compare series with line or with one view each. Only three series get a colour; past that they are drawn in neutral ink and the legend says so, so prefer a topKBy before a coloured chart. And a column the rows do not have is reported on the chart, not silently swapped — if you see that notice, the step before the view is not producing what you think it is.

Rules that determine whether the analysis is auditable

1. Name every step in the user's language. "Drop payments and credits", not "filter1". The name is what they read in the summary and the graph.

2. One concern per step. The most common mistake is fusing everything into one map:

// BAD — a black box. If the total is wrong, nothing shows which part broke.
.map("Process", (r) => ({ month: parseDate(r), amount: clean(r.Debit) }))

// GOOD — each stage's output is inspectable on its own.
.filter("Drop payments and credits", (r) => r.Debit !== "")
.map("Normalize Citi charges", (r) => ({ month: …, amount: … }))

Splitting costs nothing and is the difference between "the number is wrong somewhere" and "step 3 turned every amount into zero".

3. Filter in its own step, never inside a map. Dropped rows are frequently the bug. As a step, the drop is counted and visible.

4. Never do real work outside a step. A plain loop between steps is invisible, and its output looks like it appeared from nowhere.

5. Prefer flatMap over apply. apply cannot track which input row produced which output row, so anything downstream of it traces back only approximately. Parsing a file into records is a flatMap.

6. Normalize each source in its own branch, then union. Two statements with different schemas get two normalize steps. This is what makes a per-source bug obvious — one branch showing all zeros next to another branch that looks fine is instantly diagnosable.

7. Keep the source value beside the derived one. When you parse "1,234.56" into 1234.56, carry the original along. The deep-dive table then shows both and the user can check the conversion by eye instead of trusting it.

8. Carry money as integer cents. Sum floats and the chart says 882.1100000000001, which reads as a bug in your arithmetic even when the answer is right — and a total that disagrees with its own rows in the last decimal place destroys exactly the trust you are here to build. Convert once in the normalize step (Math.round(Number(x) * 100)), sum integers, divide by 100 in a final map before the chart.

9. Never mutate a row. Return a new object from map; don't assign to the row you were handed. Steps record rows by reference, so mutating one rewrites what an earlier step already recorded, and the deep dive will show that step producing values it never produced.

10. Prefer many small steps. Steps are cheap. Auditability is not.

11. Check the number you are going to quote. Write a .check() that traces your headline number back to the raw rows and re-adds them (see below). The verification then lives in the artifact rather than in your head, which is the whole point of the artifact. An analysis with no checks is one where the user has to take your word for the part you were most confident about.

12. End every branch with .chart() or .table(). A forty-step graph is mostly plumbing, and only you know which steps are the answer — the viewer lists views first and tints them differently precisely because it will not guess. A branch that stops at a sort is a result you left unlabelled, and the user has to open steps until they find it. If a branch's last step is not worth showing them, it probably should not be in the script.

Check your own work before reporting

report() prints every step with its row count. Read it. Investigate before you report if you see:

• a step whose row count is 0, or far below the input — a filter or join predicate that matched nothing
• a filter that dropped nothing — a predicate that matched everything
• a join with a high unmatchedLeft in its meta — keys that don't line up
• a step flagged lineage: approximate — an apply you could have written as map/flatMap
• a normalize step producing constant values (all 0, all NaN, all the same month) — a parse that silently failed
• two sources treated differently. If one branch drops payments and refunds and the other doesn't, the two statements are being counted on different rules. Check every branch for the filter its own schema needs; a signed amount column hides its refunds inside the number.

Then spot-check the actual data. Step outputs are plain JSON:

.source-and-sum/runs/<runId>/steps/s3.json

Read one or two directly. Look at whether the numbers are plausible and whether any column is uniformly empty.

Verify the headline number, in the script

Write a check for the number you are going to quote. Put it before report(), so the verification lands in the artifact next to the chart it is about — the user opening the viewer then sees that you checked, what you asserted, and that it held. A verification you did in your head, or in a console.log, reaches nobody.

behind.rowsBehind(i) walks row i of this collection back to the raw parsed records underneath it. Re-derive the number from the source columns there, not from anything you computed on the way:

monthly.check("Newest month matches the statement rows", (rows, behind) => {
  const raw = behind.rowsBehind(0) as Row[];    // row 0 of the chart — the bar you'll quote
  const cents = raw.reduce((sum, r) => sum + Math.round(Number(r.Debit || r.Amount) * 100), 0);
  return {
    ok: cents === Math.round(rows[0]!.value * 100),
    detail: `charted $${rows[0]!.value.toFixed(2)}, ${raw.length} rows sum to $${(cents / 100).toFixed(2)}`,
  };
});

Always return the detail. "Charted $882.11, 13 rows sum to $882.11" is the sentence that makes the check worth having; a bare true says only that something was tested.

A failed check does not throw — the artifact is exactly what you need when one fails. It is recorded, the run finishes, and report()'s summary prints the failure. Set the exit code yourself if the script is going to be re-run:

await a.report();
if (a.checks.some((c) => c.verdict === "failed")) process.exitCode = 1;

Then read the summary and act on it:

• A check failed. Your pipeline is wrong. Find it before you report anything; do not quote the number and mention the check.
• approximate on a check (the summary says "over an approximate trace"). An apply is in the path, so the check compared against a superset of the rows behind the number. It verified nothing. Rewrite the apply as map/flatMap, or say plainly in your report that the number is not verified.
• unchecked. The run was in production mode; nothing was checked at all.

Other things worth a check, cheaply: that two rollups of the same data agree (a leftJoin's "Uncategorized" bucket against the difference that found the same merchants), that a total equals the sum of its parts, that a lookup table has no duplicate keys (distinct then compare lengths — a duplicate silently doubles every row it matches in a join).

The same trace is available from a separate script against a finished run, which is what the visualizer does:

import { readFileSync } from "node:fs";
import { buildGraph, traceToOrigins } from "@source-and-sum/core/graph";

const dir = JSON.parse(readFileSync(".source-and-sum/latest.json", "utf8")).dir;
const read = (p: string) => JSON.parse(readFileSync(`${dir}/${p}`, "utf8"));
const manifest = read("manifest.json");
// Traced runs only — a production run writes no lineage files.
const lineages = Object.fromEntries(
  manifest.steps.map((s: any) => [s.id, read(`lineage/${s.id}.json`)]),
);

const step = manifest.steps.find((s: any) => s.name === "Monthly spending");
const { contributions } = traceToOrigins((manifest, lineages), step., []);
 ( [stepId, rows]  contributions) {
  .(stepId, rows., );
}

Report back like this

Give the answer first. Then tell the user how to check it — as a link, not as directions:

Total 2026 spending: $4,308.11 across 63 charges from 2 statements. The newest month traces back to 13 statement rows that sum to the same $882.11.

npx source-and-sum view

If anything looks off, start at s6 Normalize USAA charges — it's where the amount column gets converted: http://127.0.0.1:5178/#2026-07-27T14-12-34/s6

The viewer takes #<runId>/<stepId> in its address, so the step you are worried about is one click away instead of something the user has to go find. Take the run id from report()'s return value (artifact.runId) or from .source-and-sum/latest.json. Use port 5178 unless you started the viewer on another one.

Rules for the report:

• Say what you checked and what it found. "The May total traces back to 13 statement rows that sum to the same $882.11" is the sentence that earns the number. If a check failed, do not report the number at all — fix it first. If a check came back over an approximate trace, say the number is not verified.
• Name the step you'd check first, and say why. You know where the fragile assumption is; tell them. Give its deep link.
• Call out any assumption you made about the data — a column you guessed at, a currency you assumed, rows you dropped and how many.
• If any step has approximate lineage, say so. Do not describe a number as verified when the trace through it is approximate.
• Never round a caveat away. "I couldn't categorize 12 of 340 charges" is more useful than a clean-looking total that quietly excludes them.

Recipes

A folder of files

Keep one branch per file so lineage attributes each row to its own source.

import { globSync } from "node:fs";

const all = a.union(
  "All statements",
  globSync("statements/*.csv").map((path) => {
    const name = path.split("/").pop()!;   // becomes a step name, keep it short
    return a
      .file(path)
      .flatMap("parse file", (f) => parse(f.text, { columns: true }) as Row[])
      .map(`Normalize ${name}`, normalize);
  }),
);

One branch per file, not one step that parses all of them — that is what lets a row trace back to the file it came from, and what makes a single malformed statement visible instead of averaged in.

union throws when given nothing to combine — both spellings — so a glob that matched nothing fails here rather than becoming a confident $0 three steps later. Don't catch it: check the path.

Categories via a lookup table

const categories = a.source("category lookup", [
  { merchant: "Costco", category: "Groceries" },
  { merchant: "Netflix", category: "Entertainment" },
]);

all
  .leftJoin("Attach category", categories, {
    left: (t) => t.merchant,
    right: (c) => c.merchant,
    select: (t, c) => ({ ...t, category: c?.category ?? "Uncategorized" }),
  })
  .groupBy("Group spending by category", (t) => t.category)
  .reduceGroups("Sum spending per category", (cents: number, t) => cents + t.cents, 0)
  .sort("Largest first", (x, y) => y.value - x.value)
  .map("Cents to dollars", (r) => ({ key: r.key, value: r.value /  }))
  .(, { : , : , :  });

Use leftJoin with an explicit "Uncategorized" fallback, not join — otherwise unmatched charges vanish from the total and nobody notices.

A merchant listed twice in the lookup matches twice, duplicating every charge at it — and the inflated total looks entirely reasonable. One step and one check rule it out:

categories
  .distinct("One row per merchant", (c) => c.merchant)
  .check("The lookup has no duplicate merchants", (rows) => ({
    ok: rows.length === categories.length,
    detail: `${categories.length} lookup rows over ${rows.length} merchants`,
  }));

What the lookup table missed

all
  .difference("Charges with no category", categories, {
    left: (t) => t.merchant,
    right: (c) => c.merchant,
  })
  .groupBy("Group uncharted merchants", (t) => t.merchant)
  .reduceGroups("Sum spending per uncharted merchant", (cents: number, t) => cents + t.cents, 0)
  .sort("Largest first", (x, y) => y.value - x.value)
  .map("Cents to dollars", (r) => ({ key: r.key, value: r.value / 100 }))
  .table("Merchants missing from the lookup");

The "Uncategorized" bucket in the recipe above says how much fell through; this says what did, ranked by how much it is worth fixing. Run it whenever you write a lookup table — it is the cheapest check there is on a set of rules you made up, and its row count is the one you should quote in your report.

Recurring charges and dead subscriptions

const merchants = all
  .groupBy("Group charges by merchant", (t) => t.merchant)
  .reduceGroups(
    "Summarize each merchant",
    (acc, t) => {
      acc.months.add(t.month);
      acc.total += t.cents;
      acc.amounts.push(t.cents);
      if (t.month > acc.lastMonth) acc.lastMonth = t.month;
      return acc;
    },
    // A factory, not an object: passed as a value, every merchant would share
    // one accumulator and report the total of all of them. The library throws
    // rather than let that through, but write it right the first time.
    () => ({ months: new Set<string>(), total: 0, amounts: [] as number[], lastMonth: "" }),
  )
  .map("Classify each merchant", (m) => {
    const amounts = m.value.amounts;
    const spread = Math.max(...amounts) - .(...amounts);
     {
      : m.,
      : m...,
      : m..,
      : m.. / ,
      
      : m... >=  && spread <= ,
    };
  });

merchants
  .(,  m.)
  .(,  m. < cutoff)
  .(,  y. - x.)
  .();

Two separate filters, deliberately: the user can see how many merchants were recurring before the staleness cut, which is the number that tells them whether the heuristic is sane. State the heuristic in your report — it's a guess, and they should know it is.

Production mode

If the user wants the script for repeated/scheduled use rather than inspection, it runs unchanged with tracing off:

SOURCE_AND_SUM_MODE=production node analyze.ts

Identical results, no lineage, no retained intermediates, ~1.05× plain-JS speed. Leave tracing on by default — the point is auditability.