# Source And Sum > Analyze files of tabular data (CSV, TSV, JSON exports, bank/credit-card statements, logs, metrics exports) using @source-and-sum/core, so every step is named and recorded and every number in the answer traces back to the raw rows that produced it. Use whenever the user asks to analyze, summarize, categorize, reconcile, or find patterns in data files — and especially when they want to check the work afterwards. - Skill: `tantaman/source-and-sum` (Agent Skill) - Install (CLI): `npx skillmds@latest add tantaman/source-and-sum` - Raw SKILL.md: https://api.skillmd.com/api/skills/tantaman/source-and-sum/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Data & Analytics - Author: tantaman (https://skillmd.com/u/tantaman) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tantaman/source-and-sum --- # source-and-sum Write the analysis with recorded steps instead of ad-hoc scripting. The user then audits your work by looking at **the data each step produced**, not by reading your code. That is the whole point. An analysis that produces the right answer but cannot be checked has failed at the thing this library exists for. ## When to use this Use it when the user wants a real answer out of real data files: spending breakdowns, log summaries, export reconciliation, metric rollups, "which of these am I still paying for". Don't use it for a one-line `grep`/`wc` question, for data small enough to just read, or for non-tabular work. Setting up a pipeline for three rows is noise. ## Setup One command, from the directory that holds the data: ```sh npm create source-and-sum ``` It writes a `package.json` if there isn't one, installs the library (and a CSV parser only if it sees CSVs), lists the data files it found, and tells you whether to write `analyze.ts` or `analyze.mjs` for the Node in this shell. It does not write the analysis — that is yours. Doing it by hand, if you prefer: ```sh # Only if there is no package.json in THIS directory. Without one, `npm install` # walks up the tree and installs somewhere you did not mean. npm init -y && npm pkg set type=module npm install @source-and-sum/core npm install csv-parse # only if the inputs are CSV ``` Use `npm` unless you can see a `pnpm-lock.yaml`, `yarn.lock` or `bun.lockb` in the directory — a data folder usually has no package manager preference, and guessing wrong fails on a machine that has only npm. **Which file extension.** `node analyze.ts` works on Node ≥22.18, which strips types natively. Check with `node --version` before writing the file; on anything older write `analyze.mjs` in plain JS, because a `.ts` file that Node cannot run fails with an error that looks like a problem with the library. Never modify the user's source files. Read them; write only the artifact. The artifact contains the user's actual data — statement rows, salaries, whatever they gave you. If the folder is a git repository, add `.source-and-sum/` to `.gitignore` before running, and say that you did. ## The shape of a script ```ts import { analysis } from "@source-and-sum/core"; import { parse } from "csv-parse/sync"; const a = analysis("Spending 2026"); const citi = a .file("statements/citi.csv") .flatMap("parse file", (f) => parse(f.text, { columns: true }) as Row[]) .filter("Drop payments and credits", (r) => r.Debit !== "") .map("Normalize Citi charges", (r) => ({ month: `${r.Date.slice(6, 10)}-${r.Date.slice(0, 2)}`, merchant: merchantOf(r.Description), cents: Math.round(Number(r.Debit) * 100), raw: r.Debit, // keep the source value next to the derived one })); citi .groupBy("Group charges by month", (t) => t.month) .reduceGroups("Sum spending per month", (cents: number, t) => cents + t.cents, 0) .sort("Newest month first", (x, y) => y.key.localeCompare(x.key)) .map("Cents to dollars", (r) => ({ key: r.key, value: r.value / 100 })) .chart("Monthly spending", { type: "bar", x: "key", y: "value" }); await a.report(); ``` `report()` prints a step summary and writes `.source-and-sum/runs//`. If any step can throw, put the pipeline in a `try` and `report()` in the `finally` — a run that dies without an artifact is the one you most needed to look at. **Re-running after a fix keeps both runs.** Each `report()` writes a new timestamped directory; the newest 30 are kept. That is what lets the user ask whether your fix changed the step, so do not pass `runId` to overwrite the previous run. Do pass a label — it is what the run picker shows, and `2026-07-27T14-12-34` on its own says nothing about what you changed: ```ts await a.report({ label: "amount column parsed as cents" }); ``` ## Primitives | | | | --- | --- | | `a.file(path)` | one row: `{ path, name, bytes, text }` | | `a.source(name, rows)` | in-memory rows (lookup tables, constants) | | `.map(name, (row, i) => out)` | 1:1 | | `.filter(name, (row, i) => bool)` | keep rows | | `.flatMap(name, (row, i) => out[])` | 1:N — use for parsing | | `.sort(name, (a, b) => n)` / `.topK(name, k, cmp)` | ordering | | `.topKBy(name, k, (row) => key, "highest" \| "lowest")` | top N — **prefer this over `topK`** | | `.distinct(name, (row) => key)` | one row per key, keeping the first | | `.groupBy(name, (row) => key)` | → `{ key, items }[]` | | `.reduce(name, (acc, row, i) => acc, init)` | → one row | | `.reduceGroups(name, (acc, item, i) => acc, init)` | → `{ key, value }[]`; `init` **must** be a factory (`() => ({…})`) for a mutable accumulator | | `.join(name, right, { left, right, select })` / `.leftJoin(…)` | `select` gets `undefined` on the right for unmatched rows in a left join | | `.union(name, ...others)` | concatenate a fixed set; types may differ | | `a.union(name, collections)` | concatenate a list built at runtime | | `.intersect(name, right, { left, right })` | keep this collection's rows whose key is also on the right | | `.difference(name, right, { left, right })` | keep this collection's rows whose key is **not** on the right | | `.apply(name, (rows) => out[])` | escape hatch — **breaks row-level lineage** | | `.check(name, (rows, behind) => ok)` | record a claim and whether it held — see below | | `.chart(name, { type, x, y, color })` / `.table(name)` | mark a result — see the marks below | Steps run eagerly, so `collection.rows` is real data at any point — log it while you are working. **Use `topKBy`, not `topK`.** A comparator can be written backwards, and when it is, you get the *bottom* five under a step you named "Top 5 categories" — real rows, exact lineage, a chart that agrees with them, and nothing anywhere to tell you. `.topKBy("Top 5 categories", 5, (r) => r.value, "highest")` cannot be inverted by accident. Reach for `topK` only when the ranking genuinely needs a multi-key comparator. Ranking keys must be a number, string, bigint, boolean or Date, and every row must give the same kind — if a row's key can be missing or `NaN`, filter those rows in their own step first (you want that count visible anyway). `distinct` keeps the first row carrying each key: `"unique merchants"`, `"one row per id"`. Prefer it over `groupBy` + `map` when you want the rows rather than the groups, and never reach for `apply` to deduplicate — `distinct` keeps exact lineage and `apply` throws it away. Its lineage names every row with the key, so tracing a merchant row shows you all its charges, and `meta.removed` tells you how many duplicates there were. `intersect` and `difference` keep *this* collection's rows; the right-hand side only decides which of them survive, and contributes no values. Reach for them where you would otherwise build an array outside a step and filter against it (rule 4) — "charges at merchants the lookup has never heard of" is a `difference`, and written that way the lookup is a visible input with a row count, so a lookup that arrived empty is on screen instead of silently removing nothing. Neither one deduplicates: two rows with the same key both survive. ## Marks A chart names columns and what they mean. It never sets a scale, a domain, a format, or a colour — those are derived from the rows, which is what keeps the picture and the table under it from disagreeing. | Mark | What it is for | Channels | |---|---|---| | `bar` | ranking one measure across categories | `x` category, `y` magnitude | | `line` | a measure over time | `x` position, `y` magnitude, `color` series | | `point` | one measure against another | `x`, `y` magnitudes, `color` identity | | `cell` | two categories indexing one number (a heatmap) | `x`, `y` categories, `color` magnitude | | `table` | the rows alone | — | A channel is a column name, or `{ field, type, title }` when you need to correct the inferred type (`"quantitative" | "nominal" | "temporal"`) or label an axis: ```ts .chart("Spending by month and category", { type: "cell", x: { field: "month", type: "temporal" }, y: "category", color: { field: "dollars", title: "spent" }, }); ``` Three things the viewer will do that are worth knowing before you are surprised by them. **`bar` ignores `color`** — a set of nominal bars is one series; compare series with `line` or with one view each. **Only three series get a colour**; past that they are drawn in neutral ink and the legend says so, so prefer a `topKBy` before a coloured chart. And **a column the rows do not have is reported on the chart**, not silently swapped — if you see that notice, the step before the view is not producing what you think it is. ## Rules that determine whether the analysis is auditable **1. Name every step in the user's language.** `"Drop payments and credits"`, not `"filter1"`. The name is what they read in the summary and the graph. **2. One concern per step.** The most common mistake is fusing everything into one `map`: ```ts // BAD — a black box. If the total is wrong, nothing shows which part broke. .map("Process", (r) => ({ month: parseDate(r), amount: clean(r.Debit) })) // GOOD — each stage's output is inspectable on its own. .filter("Drop payments and credits", (r) => r.Debit !== "") .map("Normalize Citi charges", (r) => ({ month: …, amount: … })) ``` Splitting costs nothing and is the difference between "the number is wrong somewhere" and "step 3 turned every amount into zero". **3. Filter in its own step, never inside a map.** Dropped rows are frequently the bug. As a step, the drop is counted and visible. **4. Never do real work outside a step.** A plain loop between steps is invisible, and its output looks like it appeared from nowhere. **5. Prefer `flatMap` over `apply`.** `apply` cannot track which input row produced which output row, so anything downstream of it traces back only approximately. Parsing a file into records is a `flatMap`. **6. Normalize each source in its own branch, then `union`.** Two statements with different schemas get two normalize steps. This is what makes a per-source bug obvious — one branch showing all zeros next to another branch that looks fine is instantly diagnosable. **7. Keep the source value beside the derived one.** When you parse `"1,234.56"` into `1234.56`, carry the original along. The deep-dive table then shows both and the user can check the conversion by eye instead of trusting it. **8. Carry money as integer cents.** Sum floats and the chart says `882.1100000000001`, which reads as a bug in your arithmetic even when the answer is right — and a total that disagrees with its own rows in the last decimal place destroys exactly the trust you are here to build. Convert once in the normalize step (`Math.round(Number(x) * 100)`), sum integers, divide by 100 in a final `map` before the chart. **9. Never mutate a row.** Return a new object from `map`; don't assign to the row you were handed. Steps record rows by reference, so mutating one rewrites what an *earlier* step already recorded, and the deep dive will show that step producing values it never produced. **10. Prefer many small steps.** Steps are cheap. Auditability is not. **11. Check the number you are going to quote.** Write a `.check()` that traces your headline number back to the raw rows and re-adds them (see below). The verification then lives in the artifact rather than in your head, which is the whole point of the artifact. An analysis with no checks is one where the user has to take your word for the part you were most confident about. **12. End every branch with `.chart()` or `.table()`.** A forty-step graph is mostly plumbing, and only you know which steps are the answer — the viewer lists views first and tints them differently precisely because it will not guess. A branch that stops at a `sort` is a result you left unlabelled, and the user has to open steps until they find it. If a branch's last step is not worth showing them, it probably should not be in the script. ## Check your own work before reporting `report()` prints every step with its row count. Read it. Investigate before you report if you see: - a step whose row count is **0**, or far below the input — a filter or join predicate that matched nothing - a `filter` that dropped **nothing** — a predicate that matched everything - a `join` with a high `unmatchedLeft` in its meta — keys that don't line up - a step flagged `lineage: approximate` — an `apply` you could have written as `map`/`flatMap` - a normalize step producing constant values (all `0`, all `NaN`, all the same month) — a parse that silently failed - **two sources treated differently.** If one branch drops payments and refunds and the other doesn't, the two statements are being counted on different rules. Check every branch for the filter its own schema needs; a signed amount column hides its refunds inside the number. Then spot-check the actual data. Step outputs are plain JSON: ``` .source-and-sum/runs//steps/s3.json ``` Read one or two directly. Look at whether the numbers are plausible and whether any column is uniformly empty. ### Verify the headline number, in the script **Write a `check` for the number you are going to quote.** Put it before `report()`, so the verification lands in the artifact next to the chart it is about — the user opening the viewer then sees that you checked, what you asserted, and that it held. A verification you did in your head, or in a `console.log`, reaches nobody. `behind.rowsBehind(i)` walks row `i` of this collection back to the raw parsed records underneath it. Re-derive the number from the *source columns* there, not from anything you computed on the way: ```ts monthly.check("Newest month matches the statement rows", (rows, behind) => { const raw = behind.rowsBehind(0) as Row[]; // row 0 of the chart — the bar you'll quote const cents = raw.reduce((sum, r) => sum + Math.round(Number(r.Debit || r.Amount) * 100), 0); return { ok: cents === Math.round(rows[0]!.value * 100), detail: `charted $${rows[0]!.value.toFixed(2)}, ${raw.length} rows sum to $${(cents / 100).toFixed(2)}`, }; }); ``` Always return the `detail`. "Charted $882.11, 13 rows sum to $882.11" is the sentence that makes the check worth having; a bare `true` says only that something was tested. A failed check does **not** throw — the artifact is exactly what you need when one fails. It is recorded, the run finishes, and `report()`'s summary prints the failure. Set the exit code yourself if the script is going to be re-run: ```ts await a.report(); if (a.checks.some((c) => c.verdict === "failed")) process.exitCode = 1; ``` Then read the summary and act on it: - **A check failed.** Your pipeline is wrong. Find it before you report anything; do not quote the number and mention the check. - **`approximate` on a check** (the summary says "over an approximate trace"). An `apply` is in the path, so the check compared against a superset of the rows behind the number. It verified nothing. Rewrite the `apply` as `map`/`flatMap`, or say plainly in your report that the number is not verified. - **`unchecked`.** The run was in production mode; nothing was checked at all. Other things worth a check, cheaply: that two rollups of the same data agree (a `leftJoin`'s `"Uncategorized"` bucket against the `difference` that found the same merchants), that a total equals the sum of its parts, that a lookup table has no duplicate keys (`distinct` then compare lengths — a duplicate silently doubles every row it matches in a join). The same trace is available from a separate script against a finished run, which is what the visualizer does: ```ts import { readFileSync } from "node:fs"; import { buildGraph, traceToOrigins } from "@source-and-sum/core/graph"; const dir = JSON.parse(readFileSync(".source-and-sum/latest.json", "utf8")).dir; const read = (p: string) => JSON.parse(readFileSync(`${dir}/${p}`, "utf8")); const manifest = read("manifest.json"); // Traced runs only — a production run writes no lineage files. const lineages = Object.fromEntries( manifest.steps.map((s: any) => [s.id, read(`lineage/${s.id}.json`)]), ); const step = manifest.steps.find((s: any) => s.name === "Monthly spending"); const { contributions } = traceToOrigins(buildGraph(manifest, lineages), step.id, [0]); for (const [stepId, rows] of contributions) { console.log(stepId, rows.length, "contributing rows"); } ``` ## Report back like this Give the answer first. Then tell the user how to check it — as a link, not as directions: > Total 2026 spending: $4,308.11 across 63 charges from 2 statements. The > newest month traces back to 13 statement rows that sum to the same $882.11. > > ```sh > npx source-and-sum view > ``` > > If anything looks off, start at **s6 `Normalize USAA charges`** — it's where > the amount column gets converted: > The viewer takes `#/` in its address, so the step you are worried about is one click away instead of something the user has to go find. Take the run id from `report()`'s return value (`artifact.runId`) or from `.source-and-sum/latest.json`. Use port 5178 unless you started the viewer on another one. Rules for the report: - **Say what you checked and what it found.** "The May total traces back to 13 statement rows that sum to the same $882.11" is the sentence that earns the number. If a check failed, do not report the number at all — fix it first. If a check came back over an approximate trace, say the number is *not* verified. - Name the step you'd check first, and say why. You know where the fragile assumption is; tell them. Give its deep link. - Call out any assumption you made about the data — a column you guessed at, a currency you assumed, rows you dropped and how many. - If any step has approximate lineage, say so. Do not describe a number as verified when the trace through it is approximate. - Never round a caveat away. "I couldn't categorize 12 of 340 charges" is more useful than a clean-looking total that quietly excludes them. ## Recipes ### A folder of files Keep one branch per file so lineage attributes each row to its own source. ```ts import { globSync } from "node:fs"; const all = a.union( "All statements", globSync("statements/*.csv").map((path) => { const name = path.split("/").pop()!; // becomes a step name, keep it short return a .file(path) .flatMap("parse file", (f) => parse(f.text, { columns: true }) as Row[]) .map(`Normalize ${name}`, normalize); }), ); ``` One branch per file, not one step that parses all of them — that is what lets a row trace back to the file it came from, and what makes a single malformed statement visible instead of averaged in. `union` throws when given nothing to combine — both spellings — so a glob that matched nothing fails here rather than becoming a confident `$0` three steps later. Don't catch it: check the path. ### Categories via a lookup table ```ts const categories = a.source("category lookup", [ { merchant: "Costco", category: "Groceries" }, { merchant: "Netflix", category: "Entertainment" }, ]); all .leftJoin("Attach category", categories, { left: (t) => t.merchant, right: (c) => c.merchant, select: (t, c) => ({ ...t, category: c?.category ?? "Uncategorized" }), }) .groupBy("Group spending by category", (t) => t.category) .reduceGroups("Sum spending per category", (cents: number, t) => cents + t.cents, 0) .sort("Largest first", (x, y) => y.value - x.value) .map("Cents to dollars", (r) => ({ key: r.key, value: r.value / 100 })) .chart("Spending by category", { type: "bar", x: "key", y: "value" }); ``` Use `leftJoin` with an explicit `"Uncategorized"` fallback, not `join` — otherwise unmatched charges vanish from the total and nobody notices. A merchant listed twice in the lookup matches twice, duplicating every charge at it — and the inflated total looks entirely reasonable. One step and one check rule it out: ```ts categories .distinct("One row per merchant", (c) => c.merchant) .check("The lookup has no duplicate merchants", (rows) => ({ ok: rows.length === categories.length, detail: `${categories.length} lookup rows over ${rows.length} merchants`, })); ``` ### What the lookup table missed ```ts all .difference("Charges with no category", categories, { left: (t) => t.merchant, right: (c) => c.merchant, }) .groupBy("Group uncharted merchants", (t) => t.merchant) .reduceGroups("Sum spending per uncharted merchant", (cents: number, t) => cents + t.cents, 0) .sort("Largest first", (x, y) => y.value - x.value) .map("Cents to dollars", (r) => ({ key: r.key, value: r.value / 100 })) .table("Merchants missing from the lookup"); ``` The `"Uncategorized"` bucket in the recipe above says *how much* fell through; this says *what* did, ranked by how much it is worth fixing. Run it whenever you write a lookup table — it is the cheapest check there is on a set of rules you made up, and its row count is the one you should quote in your report. ### Recurring charges and dead subscriptions ```ts const merchants = all .groupBy("Group charges by merchant", (t) => t.merchant) .reduceGroups( "Summarize each merchant", (acc, t) => { acc.months.add(t.month); acc.total += t.cents; acc.amounts.push(t.cents); if (t.month > acc.lastMonth) acc.lastMonth = t.month; return acc; }, // A factory, not an object: passed as a value, every merchant would share // one accumulator and report the total of all of them. The library throws // rather than let that through, but write it right the first time. () => ({ months: new Set(), total: 0, amounts: [] as number[], lastMonth: "" }), ) .map("Classify each merchant", (m) => { const amounts = m.value.amounts; const spread = Math.max(...amounts) - Math.min(...amounts); return { merchant: m.key, monthsCharged: m.value.months.size, lastMonth: m.value.lastMonth, total: m.value.total / 100, // Charged in 3+ distinct months at a near-constant amount (within $1). recurring: m.value.months.size >= 3 && spread <= 100, }; }); merchants .filter("Keep recurring charges", (m) => m.recurring) .filter("Keep ones with no charge in the last 2 months", (m) => m.lastMonth < cutoff) .sort("Largest total first", (x, y) => y.total - x.total) .table("Possibly dead subscriptions"); ``` Two separate filters, deliberately: the user can see how many merchants were recurring before the staleness cut, which is the number that tells them whether the heuristic is sane. State the heuristic in your report — it's a guess, and they should know it is. ## Production mode If the user wants the script for repeated/scheduled use rather than inspection, it runs unchanged with tracing off: ```sh SOURCE_AND_SUM_MODE=production node analyze.ts ``` Identical results, no lineage, no retained intermediates, ~1.05× plain-JS speed. Leave tracing on by default — the point is auditability.