Missing Not at Random: when the blanks aren't an accident, your average lies

Before you start

New to this? A missing value is a blank cell — someone didn’t answer, a sensor dropped a reading, a student was absent. In pandas a blank number reads as NaN (“not a number”). You have to decide what to do with it, and there are three usual moves: drop the rows with blanks, fill the blanks with some value (a 0, or the column’s average), or ignore the problem and hope. Here’s the trap this project is about: those three choices can give you three very different answers, and the tidy “just drop the blanks” is often the most wrong. It goes wrong precisely when the blanks aren’t random — when who is missing depends on what their value would have been. Analysts call that missing not at random (MNAR). Tiny example: ask a room to rate their mood 0–10, and the grumpiest people quietly don’t answer. Average whoever did answer and you’ll report a cheerier room than the one you’re standing in. This project teaches you to catch that before it reaches a slide.

1. The Brief

You’re handed 400 rows from this year’s employee engagement survey and asked one innocent question: what’s our average engagement? One line of pandas gives you a confident, healthy number — and it’s nearly seven points too high, because the least-engaged people are exactly the ones who left the score blank, and dropping the blanks quietly deleted them from the average. Swing the other way and fill every blank with a 0 and you’re now thirteen points too low. Same data, a twenty-point spread, and nothing in the numbers you kept tells you which answer is right. This is missing-not-at-random bias, and it shows up wherever people opt out of being measured: survey scores, star ratings, no-shows, churned customers who stop answering. You’ll quantify the missingness, watch the answer move under each way of handling it, and learn to name the direction of the bias instead of being surprised by it.

2. Difficulty & time

Difficulty 1/10. One tool (pandas), one file, one concept, and Part A’s path is fully specified — the floor of the scale. It sits below the difficulty-3 data-analytics projects silent-row-loss (validate a join), benford-fraud (leading-digit screening), and cohort-retention (build a cohort triangle): each of those has more moving parts — a join to audit, a distribution to compare, a matrix to pivot — whereas this turns on a single idea (missing values that depend on their own size bias the average) applied to one column. It is a touch simpler than averaging-averages (2), which asks you to reconstruct a weighted mean; here Part A only asks you to see four numbers disagree and reason about why. It stays off the absolute bottom because it is a genuine judgment skill — knowing to distrust the tidy answer — not a syntax drill. Part A is a short read-and-run; Part B is where you do the work.

3. What you’ll be able to do after

Quantify missingness in a column (how many blanks, and what fraction) before computing anything from it.
Show how dropna, fillna(0), and fillna(mean) give different averages from the same data, and say which direction each one leans.
Recognise “missing not at random” — when the blanks are correlated with the value — and explain why dropping them biases the average, in one sentence.
Produce a defensible estimate by conditioning on a fully-observed column, and quantify how far the naive answer was off.

4. Prereqs & time box

Part A: ~30 minutes. Part B: ~1.5 hours. Both are hard caps — if Part A runs long, you’re overthinking it. You need to read a CSV with pandas and take the mean of a column; nothing more. The only new idea is NaN (a blank cell) and the three things you can do about it.

For where to write and run Python (install, virtual environment, per-OS terminal, running a script step by step), see Start Here guide. No account and no GPU — requires_signup is empty.

5. Setup & data

The data is synthetic, generated locally by data/make_data.py from a fixed seed (byte-identical every run) — see Sources. Generate it once:

python data/make_data.py

That writes data/survey_a.csv (Part A) and data/test_b.csv (Part B). Nothing is downloaded; there is no external source, and there is no real person or PII in it. Synthetic is the right call here — and the only honest one: the whole lesson is “does your average match the true average of everyone, including the people who left the answer blank,” and you can only check that when the hidden values are known in advance. The generator injects the missingness on purpose (the least-engaged skip the score; struggling students skip test day) and documents it.

6. Part A — Guided Build

Run the guided script (or paste it cell-by-cell — each # %% is one step):

python part_a/analyze_engagement.py

It walks seven steps and writes part_a/engagement_summary_a.csv. The shape of the argument:

STEP 1–2. Load the 400 rows and quantify the missingness first. Checkpoint: 118 scores are blank — a 70.5% response rate. Nearly a third are silent; that’s too many to ignore, and the only question is whether the silence is random.
STEP 3. Drop the blanks and average the rest — the one-liner everyone reaches for. Checkpoint: about 69.81. Looks like a healthy, engaged workforce.
STEP 4. Fill the blanks with 0 instead (“silence means checked out”). Checkpoint: about 49.22 — twenty points lower, from the same data.
STEP 5. Fill the blanks with the observed mean — a popular “safe” default. Checkpoint: about 69.81, identical to the dropna answer. Filling with the mean can’t move the average; it just launders the same biased number.
STEP 6. The reveal: the true average of all 400 employees is 62.89 (we know only because the data is synthetic). Dropna is +6.9 too high; fill-with-0 is 13.7 too low. The blanks hid the low scores, so dropping them floated the average up.

Why this and not that

Why not just df['engagement'].dropna().mean()? Because it silently answers “the average among people who chose to answer” — and the people who chose to answer are the more engaged ones. When missingness depends on the value (MNAR), the rows you keep no longer represent the rows you lost, so the average drifts in a predictable direction.
Why doesn’t “fill with the mean” fix it? Because the mean you’d fill with is the observed mean — already biased high. Filling blanks with it adds nothing new and leaves the average exactly where dropna left it. A fix has to bring in information the blanks don’t contain (Part B does exactly that).
Why can’t you just detect this from the data? You can’t — that’s the danger. Nothing in the 282 scores you kept warns you they’re a skewed sample. You catch it by reasoning about who is missing and why, not by staring at the numbers.

Functions you’ll meet (and where to read more)

Function	What it does here
`pandas.read_csv`	Load `survey_a.csv`; blank cells become `NaN`
`Series.isna`	Flag the missing scores so you can count them (STEP 2)
`Series.dropna`	Drop the blanks before averaging — the biased move (STEP 3)
`Series.fillna`	Fill the blanks with 0 or the mean (STEP 4–5)
`Series.mean`	The average, computed each way

If something looks off

If all four numbers agree, your blanks are being read as something other than NaN (a literal string, say) — check survey["engagement"].isna().sum() is 118, not 0.
If dropna comes out below the true mean instead of above, your missingness would have to be hiding the high scores — the direction of the bias always follows who went missing.
A number that surprises you is the point of this project, not a bug — read it against the checkpoints above before assuming you broke something.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already writes the naive (biased) answer; your job is to finish the one # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Same skill, new setting. A school district’s test_b.csv has 500 students, each with a prior_band from last term (Low / Mid / High, filled in for everyone) and a test_score that is blank for the students who were absent on test day. The catch: Low-band students were far more likely to be absent, so the present-only average overstates the cohort. The starter writes that naive answer — the mean of whoever showed up. The # TODO is the transfer from Part A: don’t just drop the absentees. Because within a band the absences are essentially random, you can estimate each band’s average from its present students and put the missing ones back — weighting each band by how many students it has in total, which you know from prior_band.

Acceptance criteria (checkable)

Your part_b/starter.py writes part_b/out/report.json with:

estimated_mean — your debiased estimate of the whole-cohort average test score,
n_students — the total number of students, present and absent.

Then:

python part_b/starter.py
pytest part_b/test_solution.py -q

You’re done when the tests pass: n_students is exact (500) and estimated_mean lands within 1% of the true whole-cohort average (the band-stratified reference lands within ~0.05%). The naive present-only mean is about 3.5 points — over 5% — too high and fails.

Constraints — what must be true of your solution, not how to get there

The estimate must account for every student, including the absent ones — not just who showed up.
Don’t invent scores for individual absentees; work at the band level, where the present students are a fair sample.
prior_band is your lever because it is observed for everyone. If you don’t use it, you have no way to put the missing students back.

Hints

Hint 1 — why the naive answer leans high

The absent students are concentrated in the Low band, which scores lowest. Dropping them removes mostly low scores, so the average of who’s left floats up — the same mechanism as Part A’s dropna, just with “absent” instead of “left it blank.”

Hint 2 — what "within a band it's random" buys you

Overall, missingness depends on the score (Low scorers skip more) — that’s the bias. But hold the band fixed and the absences are random, so the students present in a band are a fair sample of that band. That means each band’s present-mean is a good estimate of that band’s true mean. Compute it per band with groupby("prior_band").

Hint 3 — the shape of the answer

For each band: present_mean = grp["test_score"].dropna().mean(). Weight each band by its full size (present + absent) over the total, and sum: estimate = Σ present_mean_band × (n_band / n_total). Because prior_band is filled for everyone, n_band counts absentees too — that’s what pulls the estimate back down.

8. Self-check

Your debiased estimate should come out below the naive present-only mean, because putting the absent Low-band students back drags the average down — sanity-check the direction.
Re-running python data/make_data.py never changes the data (it’s seeded), so your number shouldn’t move between runs.
If you can state, in one sentence, why dropping the blanks was biased and what observed column let you fix it, you’ve got the transferable skill — the specific numbers are secondary.

9. Stretch

Add the per-department response rates to Part A (groupby("department")) and check whether the missingness is worse in some departments than others — a real report would break the bias down, not just state it once.
In Part B, compute a range instead of a point estimate: what’s the cohort average if every absentee scored at the bottom of their band versus the top? The gap between those two is an honest bound on how much you don’t know.
(Genuinely hard.) The band fix works because absence is random within a band. Break that assumption: modify make_data.py so that within the Low band, the very lowest scorers are also the most likely to be absent. Now band-stratifying is still biased. Show it, and reason about what extra information (beyond prior_band) you’d need to fix it — this is the line between MNAR you can repair and MNAR you can’t.

10. Ship it

Put this in a repo README so it counts as a portfolio piece: one sentence on the trap (“dropping non-random blanks overstated the engagement score by ~7 points”), the four numbers side by side (true / dropna / fill-0 / fill-mean), and the one-line rule — quantify the missingness and ask whether it’s random before you trust any average. Add a short note on Part B showing you transferred the same idea to a different domain (test scores, not survey scores) and produced a debiased estimate by conditioning on an observed column. That last part — recovering the same skill under a different surface — is what an interviewer is actually checking for.

11. Sources

See SOURCES.md. The data is synthetic, generated by data/make_data.py from a fixed seed; there is no external source, no real person or PII, and nothing is downloaded or committed.

▶ Run Part A in your browser

▶ Try it now — run the real Part A right here in your browser. This one button makes the data and runs Part A for you — you do not need to install anything, and you do not need to type any of the python … commands anywhere on this page (those are only for running on your own computer instead). First run loads Python + this project’s libraries: usually ~10–20 s, a little longer for machine-learning projects.

Next up

Finished this one? Continue the Data Analytics track:

Averaging Averages: the company number isn’t the mean of the branch numbers → · Browse all courses