Benford's Law: catching fabricated numbers by their leading digit

Data: both parts use synthetic data, generated locally by data/make_data.py (seeded, reproducible). Nothing is downloaded; no signup, no GPU. Synthetic is essential here: to score a fraud detector you must know which records were fabricated, and only data you generated gives you that ground truth.

Before you start

New to this? Two minutes of vocabulary, because the whole project turns on it.

Leading digit — the first non-zero digit of a number. The leading digit of 342.19 is 3; of 0.087 is 8; of 1,204 is 1.
Benford’s Law — a surprising fact about real-world numbers that span many sizes (expenses, populations, transaction amounts): their leading digits are not evenly spread. A 1 leads about 30% of the time, a 9 under 5%. It falls out of the fact that going from 100 to 200 (a 100% jump) covers far more of the range than 800 to 900 (a 12% jump).
Why it catches fakes — numbers a person invents (padded expenses, fabricated invoices) usually don’t follow this. People reach for uniform-ish or round figures, so their leading digits come out roughly even — a pattern that stands out against Benford.
MAD (mean absolute deviation) — one number for “how far off Benford is this batch”: average the gap between the observed digit frequencies and Benford’s. Small = conforms; large = suspicious.

A tiny example — 100 genuine expense amounts will have about 30 starting with 1 and only about 5 starting with 9. If instead you see roughly 11 of each digit, someone probably typed those numbers in by hand.

What you’ll learn: to test whether a batch of numbers follows Benford’s Law, to read the result with the right caution (it flags, it doesn’t prove), and to use it to pick fabricated accounts out of a pile of real ones.

New to Python? Section 4 (“Where to write and run code”) starts from zero.

1. The Brief

You’ll take a company’s genuine expense history and confirm its leading digits follow Benford’s Law almost exactly (a deviation of 0.007), then look at one employee’s submitted reimbursements and watch them fail the same test badly (0.055, eight times further off) because the amounts were invented. Then, in Part B, you’ll be handed an accounts-payable file with dozens of vendors, a minority of them fake, and you’ll flag the fabricated ones by their digits alone — scoring an F1 of 1.0 against the hidden truth.

This is a real forensic-accounting screen (Mark Nigrini built a career on it, and auditors run it on tax and expense data). It’s a beautiful first analytics skill because the whole thing is one honest idea — real numbers have a fingerprint, faked ones usually don’t — and because it comes with the discipline that separates analysis from superstition: Benford flags a batch for a human to examine; it never proves fraud, and it only applies to the right kind of data. Knowing when not to trust it is as much the lesson as running it.

2. Difficulty & time

Difficulty 3 / 10. It’s about level with #silent-row-loss and #idempotent-loads (both difficulty 3): a single clean idea (leading-digit conformance), one or two tools, and a Part A whose path is fully specified — but with a real judgment call (when does Benford apply?) and a fiddly debug surface (extracting the first significant digit correctly). It’s easier than #reorder-point-trap, #accuracy-is-lying, and #hidden-communities (all difficulty 5), which each add a simulation, a modelling pipeline, or a validation-metric-plus-algorithm-choice this project doesn’t need. It earns a 3 over a 1–2 because Part B is a genuine classification you must design (choose a threshold, avoid flagging everyone), not a near-neighbour of Part A. Anchored to the rubric’s level-3 band and those named ledger projects (G10).

Time is separate from difficulty: the script runs in under a second (measured Part A wall-clock 0.35s). Part A is about 45 minutes of reading and running; Part B is 2 hours of your own work.

3. What you’ll be able to do after

Compute the leading-digit distribution of a batch of numbers and compare it to Benford’s Law with a MAD statistic.
Read the result responsibly: distinguish “flag for review” from “proof,” and say when Benford’s Law does and does not apply.
Turn the test into a detector: flag fabricated accounts out of a mixed pile, and choose a sensible threshold instead of flagging everything.
Score a detector honestly with precision, recall, and F1 against known labels.

The finished result

By the end of Part A you’ll have one number — the MAD, how far a batch’s leading digits sit from Benford’s Law — that cleanly separates real spending from invented spending:

company's real expenses ...... MAD 0.0066   -> conforms (real, multi-scale data)
employee_42's reimbursements . MAD 0.0551   -> nonconforming (about 8x further off)

The genuine batch’s leading digits match Benford almost exactly; the fabricated one’s are nearly uniform — the fingerprint of numbers a person typed in. That 8× gap is the flag (a flag for a human to check, not a verdict).

4. Prereqs & time box

You can write a for loop and a function, and you’ve seen a pandas DataFrame. No statistics or accounting background is assumed — the vocabulary is in the primer above, and the one formula (Benford’s digit probabilities) is given to you. The trickiest bit is string-handling to pull the first significant digit out of a number, which the guide walks through.

Where to write and run code

Two ways to run this project — pick either:

In your browser (no install). If the project page shows a ▶ Run button, click it to run Part A right here — nothing to set up. Part A uses only pandas and numpy, which run in the browser.
On your computer. The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements, and running Part A step by step — is covered once in Start Here. Come back here once python -c "import pandas" runs without error. Part B (writing your own solution) needs this local setup.

Time box

Part A — Guided Build: 30–60 minutes. Hard cap 90.
Part B — Your Turn: 1.5–2.5 hours. Hard cap 4.

5. Setup & data

Both datasets are synthetic, generated by one seeded command (byte-for-byte reproducible, git-ignored, never committed):

python data/make_data.py

That writes data/expenses.csv (Part A — a company’s historical expenses plus one employee’s fabricated reimbursements) and data/vendor_invoices.csv (Part B — many vendors’ invoice amounts, a minority fabricated). Which records are fabricated is the ground truth, documented in data/make_data.py and used by the grader — not handed to you in Part B.

6. Part A — Guided Build

Run the full script and read along:

python data/make_data.py         # once
python part_a/check_benford.py

Prefer the browser? If this page has a ▶ Run button, click it instead of the commands above — Part A runs in your browser with nothing to install. The STEP output and every checkpoint below are identical either way.

▶ Try it now — run the real Part A right here in your browser. This one button makes the data and runs Part A for you — you do not need to install anything, and you do not need to type any of the python … commands anywhere on this page (those are only for running on your own computer instead). First run loads Python + this project’s libraries: usually ~10–20 s, a little longer for machine-learning projects.

The full, commented script is in part_a/check_benford.py.

Step 1 — Benford’s expected distribution. The probability the leading digit is d is log10(1 + 1/d).

Checkpoint: digit 1 ≈ 0.301, digit 2 ≈ 0.176, …, digit 9 ≈ 0.046.

Step 2 — The company’s historical expenses. Extract each amount’s leading digit, tally the distribution, and compare to Benford with a MAD.

Checkpoint: MAD ≈ 0.0066 — well under the 0.015 “nonconformity” line. Real, multi-scale spending follows Benford.

Step 3 — One employee’s reimbursements. Same test, different submitter.

Checkpoint: MAD ≈ 0.0551 — far above 0.015. The leading digits are nearly uniform (~0.11 each), the fingerprint of invented numbers. Red flag.

Step 4 — State it carefully. The fabricated batch deviates about 8× more than the genuine one — but Benford flags, it doesn’t convict.

Why this and not that

Why the leading digit, not the whole number or the last digit? Benford’s Law is specifically about the leading digit of multi-scale data. Last digits of many real quantities are roughly uniform, and full-number tests throw away the simple, robust signal. The leading digit is where the fingerprint lives.
Why MAD, not a chi-square test? A chi-square p-value is sample-size-dependent: with enough rows everything looks “significantly non-Benford.” MAD measures the size of the deviation, not just its statistical detectability, so it stays meaningful as batches grow — which is why forensic practice (Nigrini) uses it.
Why you must know when Benford does NOT apply. It needs numbers spanning several orders of magnitude with no built-in floor or cap. Prices capped at $9.99, ages, phone numbers, or assigned ids will “fail” Benford while being perfectly honest. Running the test where it doesn’t apply manufactures false accusations — the most important judgment in the whole project.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
`pd.read_csv`	Load the amounts	read_csv
`DataFrame.groupby(...).apply`	(Part B) compute a MAD per vendor	groupby
`np.log10`	Benford’s digit probabilities, `log10(1 + 1/d)`	log10
Concept	Benford’s Law and its use in fraud detection	Benford’s Law (Wikipedia)

If something looks off

Your Part B F1 comes out low. The usual cause is the leading-digit extraction — make sure you take the first non-zero digit (the leading digit of 0.087 is 8, not 0). Get that right and the two vendor groups separate cleanly.
A checkpoint number doesn’t match. The data is seeded, so the MADs are exact. Re-run python data/make_data.py (you may have skipped or edited it), then re-run Part A.
Setup problems (Python not found, pip, FileNotFoundError, re-activating the venv): see the troubleshooting list in Start Here.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Different data, a real detection job. data/vendor_invoices.csv is an accounts-payable file: 60 vendors, each with hundreds of invoice amounts. Most vendors are legitimate and their amounts follow Benford; a minority are shell vendors whose invoices were fabricated. You are not told which, or how many.

Your job: flag the fabricated vendors, using the leading-digit test per vendor. Same core skill as Part A (leading-digit conformance), now as a classifier over many accounts where you must choose a threshold and live with the precision/recall tradeoff.

Acceptance criteria (checkable)

Fill in part_b/starter.py, then:

python part_b/starter.py            # writes part_b/out/
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

Your solution writes into part_b/out/:

flags.csv — vendor_id, mad, flagged (1 if you think it’s fabricated).
report.json — vendors, flagged, f1, achieved_metric.

You’re done when the tests pass. The grader recovers the true fabricated vendors from the seed (you never see them) and scores your flags with F1. It requires a decision for every vendor and F1 ≥ 0.85. The reference reaches 1.00; flagging nobody or everybody fails.

Constraints — what must be true, not how to get there

Decide from the invoice amounts (their leading digits), not from vendor ids and not by reading the planted labels out of make_data.py.
Choose a threshold that separates the two groups — don’t flag every vendor to guarantee recall (that tanks precision, and F1 with it).
Your reported f1 must be the real score of your flags (the grader recomputes it).

Hints

Hint 1 — the shape of the problem

You already have the tool from Part A: leading-digit distribution → MAD against Benford. Compute that MAD per vendor (group the invoices by vendor_id), then look at the spread of MADs across vendors.

Hint 2 — choosing the threshold

Plot or sort the per-vendor MADs and you’ll see two clusters: a tight group near the conformity range (~0.01) and a separate group much higher (~0.06). Any threshold in the gap between them cleanly separates legit from fabricated — Part A’s 0.015 “nonconformity” line is a reasonable starting point.

Hint 3 — scoring yourself (still not the code)

F1 balances precision (of the vendors you flagged, how many were really fake) and recall (of the real fakes, how many you caught). Flagging everyone gives recall 1.0 but terrible precision; flagging no one gives the reverse. The clean separation here means a sensible threshold gets both near 1.0 — if your F1 is low, check your leading-digit extraction first (it’s the usual bug).

8. Self-check

You don’t need the answer key. You’re done when:

You can say why genuine expenses follow Benford and invented ones usually don’t.
You can name a kind of data where Benford’s Law would give a false alarm, and say why.
Your per-vendor MADs fall into two clear clusters, and your threshold sits in the gap between them.
Your F1 is high because both precision and recall are high — not because you flagged everything.

9. Stretch

Second-digit test. Benford also predicts the second digit’s distribution (flatter, but not uniform). Add it and see whether it catches fabrications the first-digit test misses.
Subtle fraud. Regenerate the data so a fraudster fabricates only some of a vendor’s invoices (mix real and fake). Find where the MAD signal washes out, and think about what sample size you’d need to catch partial manipulation.
(Genuinely hard) Where Benford breaks. Construct an honest dataset that fails Benford (e.g. amounts capped at an approval limit, or a single-product price list) and a fraudulent one that passes it. Show that the test alone can convict the innocent and clear the guilty — the reason it’s a screen, not a verdict.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

Lead with the one-liner anyone gets: “Real expense data starts with ‘1’ about 30% of the time; invented numbers don’t — here’s how I used that to flag fabricated vendors at F1 1.0.”
Show the bar chart of observed leading digits vs Benford for a genuine and a fabricated batch side by side — the gap is the whole story in one picture.
State the caveat in one honest sentence: Benford flags, it doesn’t prove, and it only applies to multi-scale data. Showing you know the limits is what makes the analysis trustworthy.

Keep make_data.py and the tests in the repo so anyone can reproduce your numbers from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Datasets	Synthetic — company expenses (Part A); vendor invoices (Part B)
Data type	Synthetic (seeded, reproducible)
Generated by	`data/make_data.py` — `numpy.random.default_rng` (seeds 20260718 / 4242)
Source / license	Not applicable — synthetic, generated in-repo; no third-party rights
Access / verification date	2026-07-18
Redistribution	Not applicable — generator committed, CSV output never committed
robots.txt / ToS	Not applicable — nothing is scraped or fetched

The data is synthetic and teaches the mechanic (leading-digit fraud screening); it does not describe any real company. The generative process — Benford-conforming amounts via powers of ten, fabricated amounts via a uniform range — is documented in data/make_data.py.

Next up

Finished this one? Continue the Data Analytics track:

Cohort Retention: why the topline looks healthy while every cohort leaks worse → · Browse all courses