Averaging Averages: the company number isn't the mean of the branch numbers

Before you start

New to this? A rate is one number divided by another: a profit margin is profit ÷ revenue; an average handle time is total seconds ÷ number of calls. This project is about one trap that catches nearly everyone at least once: when you have a rate for each group (each region, each agent) and you want the rate for everything together, the plain average of the group rates is usually wrong. Example: two regions, one earns £100 profit on £1,000 (10%), the other £90 on £100 (90%). The average of “10% and 90%” is 50% — but the company actually made £190 on £1,100, which is 17%. The big region should count more. This project teaches you to get that number right, every time, and to spot when someone else got it wrong.

1. The Brief

You’re handed a per-region profit-and-loss sheet and asked one innocent question: what’s our average margin this quarter? You can answer it in one line of pandas — and get a number that is almost seven points too high, because one tiny boutique with a fat margin counts as much as a region a hundred times its size. This is the “average of averages” mistake, and it shows up everywhere: blended conversion rates, portfolio yields, survey scores, SLA averages. A finance team that reports the mean of regional margins instead of the pooled margin is overstating the business, and the gap is invisible unless you know to look. You’ll build the correct number, see exactly why the naive one drifts, and learn the one rule that fixes the whole family of cases.

2. Difficulty & time

Difficulty 2/10. One tool (pandas), one file, one concept, and Part A’s path is fully specified — the low end of the scale. It sits below silent-row-loss (3), cohort-retention (3), and abc-xyz-inventory (3): each of those has more moving parts (a join to validate, a cohort matrix to pivot, two segmentation axes), whereas this turns on a single idea — weight by size — applied cleanly. It is a genuine judgment skill, not a syntax drill, which is what keeps it off the floor of the scale. Part A is a short read-and-run; Part B is where you do the work.

3. What you’ll be able to do after

Compute a group-weighted (pooled) average correctly, and explain in one sentence why it differs from the mean of the per-group figures.
Look at a reported “average of X” and say whether it should have been weighted, and by what.
Reconstruct a pooled figure when you’re only given per-group averages and their sizes (not the raw totals).
State the one condition under which a plain mean is the right answer.

4. Prereqs & time box

Part A: ~30 minutes. Part B: ~1.5 hours. Both are hard caps — if Part A runs long, you’re overthinking it. You need to read a CSV with pandas and do arithmetic on columns; nothing more.

For where to write and run Python (install, virtual environment, per-OS terminal, running a script step by step), see Start Here guide. No account and no GPU — requires_signup is empty.

5. Setup & data

The data is synthetic, generated locally by data/make_data.py from a fixed seed (byte-identical every run) — see Sources. Generate it once:

python data/make_data.py

That writes data/regions_a.csv (Part A) and data/calls_b.csv (Part B). Nothing is downloaded; there is no external source. Synthetic is the right call here: the lesson is whether your average matches the true pooled number, and only generated data lets you know that truth in advance.

6. Part A — Guided Build

Run the guided script (or paste it cell-by-cell — each # %% is one step):

python part_a/analyze_margins.py

It walks five steps and writes part_a/margins_summary_a.csv. The shape of the argument:

STEP 1–2. Load the five regions and take the naive answer — the mean of the five region margins. Checkpoint: about 0.2700 (27.0%).
STEP 3. Take the pooled answer — total profit over total revenue. Checkpoint: about 0.2025 (20.25%). Nearly seven points lower, and it’s the one that matches the bank.
STEP 4. See why: in the naive mean, the £140k “Flagship” boutique (a 55% margin) counts as one-fifth of the answer — the same as a £4.2M region — though it earned ~1% of the revenue.
STEP 5. The rule, verified in code: the revenue-weighted mean of the margins equals the pooled rate exactly. assert proves it.

Why this and not that

Why not just df['margin'].mean()? Because it answers “the average of the regions’ rates,” a question nobody asked. The company’s rate is a rate over the whole company — a weighted mean of the parts, weighted by each part’s denominator (revenue).
Why does the gap appear here and not always? Only when group sizes differ. If every region had equal revenue, the weights would be equal and the two answers would coincide. They rarely do.

Functions you’ll meet (and where to read more)

Function	What it does here
`pandas.read_csv`	Load `regions_a.csv` into a DataFrame
`Series.mean`	The naive “average of the margins” (STEP 2)
`Series.sum`	Pool profit and revenue before dividing (STEP 3)

If something looks off

If your pooled number equals your naive number, your groups are all the same size — check the revenue column actually varies.
If the naive number looks lower than pooled instead of higher, your extreme small group has a low rate, not a high one — the direction of the skew follows the small group.
A number that surprises you is the point of this project, not a bug — read it against the checkpoints above before assuming you broke something.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Same skill, new setting. A call centre’s calls_b.csv gives each agent’s average handle time and number of calls. Ops wants the team’s average handle time. The starter prints the naive answer — the mean of the agent averages — which is wrong, because two new hires with a handful of slow calls count as much as veterans with hundreds. The # TODO is the same move as Part A: weight by size. But note the twist — you’re given each agent’s average, not their total time, so you must reconstruct avg × n_calls before you can pool.

Acceptance criteria (checkable)

Your part_b/starter.py writes part_b/out/report.json with:

team_avg_handle_sec — the call-weighted team average handle time (seconds),
total_calls — the total number of calls across all agents.

Then:

python part_b/starter.py
pytest part_b/test_solution.py -q

You’re done when the tests pass: total_calls is exact and team_avg_handle_sec lands within 0.5% of the true pooled value (the reference hits it exactly). The naive mean of agent averages is about 42 seconds too high and fails.

Constraints — what must be true of your solution, not how to get there

The team average must be a single number: total handle time across all calls ÷ total calls.
Don’t drop the low-volume agents — they belong in the total; they just shouldn’t dominate it.

Hints

Hint 1 — what "weight by size" means here

In Part A the weight was revenue (the denominator of a margin). Here the denominator of “seconds per call” is the number of calls. So each agent should count in proportion to n_calls.

Hint 2 — you don't have the totals yet

You’re given avg_handle_sec, not total seconds. Total seconds for an agent is avg_handle_sec × n_calls. Sum that across agents, then divide by the summed n_calls.

Hint 3 — the shape of the answer

team_avg = (agents['avg_handle_sec'] * agents['n_calls']).sum() / agents['n_calls'].sum(). It should come out well below the naive mean, because the slow new hires barely move a call-weighted total.

8. Self-check

Your pooled team average should be lower than the naive mean (the slow agents took few calls), and close to the busy agents’ typical handle time — sanity-check it against them.
Re-running python data/make_data.py never changes the data (it’s seeded), so your number shouldn’t move between runs.
If you can state, in one sentence, what to weight by and why, you’ve got the transferable skill — the specific numbers are secondary.

9. Stretch

Add a by_region breakdown to Part A: compute each region’s contribution to the pooled margin (its margin × revenue_share) and confirm they sum to the pooled number.
Compute a confidence-aware team average in Part B: an agent with 6 calls is a noisy estimate — try shrinking each agent’s average toward the team mean in proportion to how few calls they took, and see how much the answer moves.
(Genuinely hard.) You’re given only per-group averages and counts, but you also want the pooled standard deviation of handle time — which you can’t get from means and counts alone. Work out what extra per-group statistic you’d need, and why the pooled variance isn’t the weighted mean of the group variances.

10. Ship it

Put this in a repo README so it counts as a portfolio piece: one sentence on the trap (“a plain mean of per-group rates overstated the margin by ~7 points”), the two numbers side by side, and the one-line rule. Add a short note on Part B showing you transferred the same idea to a different metric (durations, not rates) and reconstructed the totals yourself. That last part — recognising the same skill under a different surface — is what an interviewer is actually checking for.

11. Sources

See SOURCES.md. The data is synthetic, generated by data/make_data.py from a fixed seed; there is no external source, and nothing is downloaded or committed.

Next up

Finished this one? Continue the Data Analytics track:

Outlier Spotting: one bad reading wrecks the mean, not the median → · Browse all courses