Accuracy is lying to you: evaluating a classifier on imbalanced data

Project at a glance

• Track: AI & Machine Learning
• Difficulty: Intermediate · 5/10
• Time: ~1.25 h for Part A · ~3.0 h more for the optional Part B
• You’ll use: pandas, numpy, scikit-learn, pytest
• 💻 Local setup needed (no in-browser ▶ Run yet) — a one-time, free Python install, then you just run files. Start Here walks you through it.
• New here? This is an intermediate project — it assumes you can read and run a little Python. New to the setup? Start Here walks you through it.

Data: Part A uses a real dataset — the UCI Bank Marketing dataset (CC BY 4.0), downloaded by data/get_data.py and never committed. Part B uses synthetic data generated locally by data/make_data.py. Both are free, no signup, no GPU. This project assumes you’ve trained a classifier at least once; if not, see section 4.

Before you start

New to this? Two minutes of vocabulary, because the whole project turns on it.

A dataset is imbalanced when one outcome is rare — most bank customers don’t subscribe, most machines don’t fail. Accuracy is just “what fraction of predictions were right.” The trap: when 88% of cases are “no,” a model that blindly answers “no” every time is 88% accurate and completely useless.

A tiny example — 100 emails, 5 of them spam. A filter that marks everything “not spam” is 95% accurate and catches zero spam. Accuracy looked great; the filter is worthless.

So you need scores that look at the rare class specifically:

• Recall — of the things that were actually positive, what fraction did you catch? (The spam filter above: recall 0.)
• Precision — of the things you flagged as positive, what fraction really were?
• PR-AUC — one number summarising the precision/recall tradeoff across every threshold.

What you’ll learn: to stop trusting accuracy on imbalanced data, read the metrics that tell the truth, and choose the decision threshold deliberately instead of leaving it at 0.5.

New to Python too? See Start Here — it starts from zero.

1. The Brief

You’ll train a perfectly ordinary model on real data, watch it report 89.5% accuracy, and then prove that number is worthless — it’s barely above the 88.3% you’d get by predicting “no” for everyone, and it catches fewer than one in five of the customers you actually wanted to find. Then you’ll learn the scores that don’t lie, and how to move the decision threshold to the operating point your problem actually needs.

Anyone who builds or buys a classifier cares about this. “Our model is 95% accurate” is the single most common way a machine-learning result misleads the person paying for it: on rare events — fraud, disease, churn, equipment failure — high accuracy is the default, not an achievement, and a model can have it while missing almost everything that matters. Reading past accuracy is the difference between a model that works and a demo that fools a meeting.

2. Difficulty & time

Difficulty 5 / 10. It sits above the seed project #silent-row-loss (difficulty 3): it assumes you can train a classifier, uses a small scikit-learn pipeline (encoding, scaling, a model), and turns on a genuinely non-obvious idea (why PR-AUC and recall beat accuracy, and how the decision threshold is a free choice). It stays below 7–8 because there’s one tool (scikit-learn), no second system, and Part A’s path is fully specified. (Calibrated against the ledger per G10: harder than the difficulty-3 builds #silent-row-loss and #idempotent-loads, which each teach a single data-quality mechanic; about level with #reorder-point-trap (difficulty 5), which likewise chains a multi-step pipeline around a genuinely non-obvious quantitative idea.)

Time is separate from difficulty: Part A is ~75 minutes of reading and running (the script itself finishes in about 7 seconds); Part B is 2–4 hours of your own work.

3. What you’ll be able to do after

• Show why accuracy is misleading on imbalanced data, by comparing any model to a “predict the majority” baseline.
• Read a confusion matrix and compute precision, recall, and PR-AUC — and say which one matters for a given problem.
• Choose a decision threshold to hit a target (e.g. “catch 80% of failures”) instead of accepting the default 0.5, and state the precision cost of doing so.
• Spot and avoid a leaky feature (why we drop duration here) before it flatters your model.

The finished result

By the end of Part A you’ll have trained a perfectly ordinary model, watched it post a flattering accuracy, and proved that the number is hollow:

accuracy (real model) ...... 0.895   looks like a working model
"predict no" baseline ...... 0.883   catches zero subscribers — nearly the same score
recall: subscribers found .. 0.185   the model misses about four of every five you wanted
PR-AUC ..................... 0.419   the honest summary once positives are rare

Accuracy lands just 1.2 points above a model that always says “no,” and hides that the real model catches fewer than one in five actual subscribers — recall and PR-AUC are what reveal it.

4. Prereqs & time box

You can write a for loop and a function, you’ve seen a pandas DataFrame, and you’ve trained a classifier at least once (model.fit(X, y) isn’t new). The metric vocabulary is in the primer above; the ML concepts (train/test split, a logistic-regression classifier) are used but explained as we go.

Where to write and run code

Run this project on your computer. Part A downloads a real dataset from the internet (the UCI Bank Marketing dataset). The in-browser ▶ Run can’t make that network request, so this is a local-only project.

The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements, and running Part A step by step — is covered once in Start Here. Come back here once python -c "import sklearn, requests" runs without error.

Time box

• Part A — Guided Build: 60–90 minutes. Hard cap.
• Part B — Your Turn: 2–4 hours. Hard cap.

5. Setup & data

Part A (real). One command downloads the UCI Bank Marketing dataset from its official source (CC BY 4.0 — see Sources) and writes data/bank-full.csv (git-ignored, never committed):

python data/get_data.py

Part B (synthetic). One command generates the pump-fleet data (seeded, byte-for-byte reproducible), written to data/pumps.csv:

python data/make_data.py

The synthetic story: a water-treatment plant’s pumps emit sensor readings; failures are rare (~5–6%) and driven by wear (vibration, temperature, age), buried among decorative “noise” sensors. Every defect is documented in data/make_data.py. Synthetic data teaches the mechanic — it does not prove a real-world finding.

6. Part A — Guided Build

Run the whole thing, or (better) one # %% cell at a time — see Start Here (this project runs on your computer, not in the browser):

python data/get_data.py       # once
python part_a/evaluate.py

The full, commented script is in part_a/evaluate.py. Every scikit-learn function used is linked in the “Functions you’ll meet” table at the end of this section.

Step 1 — Load, and drop a leaky column. The file is ;-separated. We drop duration (the length of the call that just happened) because you’d only know it after the outcome — UCI says to discard it. Keeping it would flatter the model and hide the lesson.

df = pd.read_csv("data/bank-full.csv", sep=";").drop(columns=["duration"])
y = (df["y"] == "yes").astype(int)

Checkpoint: 45,211 rows; positive rate 0.117 (about 1 in 9 subscribed).

Step 2 — A stratified split so the test set keeps that same rare-positive rate.

Checkpoint: train 33,908 / test 11,303; test positive rate 0.117.

Step 3 — The baseline any model must beat: predict the majority with DummyClassifier.

Checkpoint: baseline accuracy 0.883 — you can be 88% “accurate” and catch zero subscribers.

Step 4 — A real model, scored by accuracy. It looks great. It isn’t. A LogisticRegression inside a Pipeline (one-hot the categoricals, scale the numbers) at the default 0.5 threshold:

Checkpoint: accuracy 0.895 — a whole 1.2 points above the do-nothing baseline (0.883). Recall 0.185: it finds fewer than one in five real subscribers. Confusion matrix: tn=9868 fp=113 fn=1078 tp=244 — of 1,322 actual subscribers it catches 244.

Step 5 — The scores that don’t lie: PR-AUC and ROC-AUC.

Checkpoint: PR-AUC 0.419, ROC-AUC 0.773. PR-AUC is anchored to the ~12% positive rate, so it tells you far more here than accuracy did.

Step 6 — Move the threshold. Accuracy froze one knob at 0.5; it’s yours to choose. Using the precision–recall curve:

Checkpoint: at threshold 0.168 you get recall 0.500 at precision 0.399 — catch half the subscribers instead of a fifth, at the cost of more false alarms. Where to stand is a business call, not a default.

Why this and not that

• Why compare to a DummyClassifier, not just report accuracy. A number means nothing without the floor beside it. 89.5% sounds like a model; next to the 88.3% floor it’s almost nothing. Always print the baseline in the same breath as the score.
• Why PR-AUC here, not ROC-AUC. ROC-AUC can look flattering when negatives dominate, because it rewards ranking the many easy negatives correctly. PR-AUC focuses on the rare positives you actually care about, so it’s the honest summary when positives are scarce.
• Why drop duration. It’s the clearest leak in this dataset: known only after the call, and near-perfectly tied to the outcome. Leaving it in would push accuracy up and quietly turn this into a different (dishonest) project.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
pd.read_csv(..., sep=";")	Loads the ;-separated bank file	read_csv
train_test_split(..., stratify=y)	Splits data, keeping the rare-class rate in both halves	train_test_split
DummyClassifier("most_frequent")	The “predict the majority” baseline to beat	DummyClassifier
LogisticRegression	A simple, honest classifier	LogisticRegression
Pipeline / ColumnTransformer	Bundle encoding + scaling + model so nothing leaks across the split	Pipeline
confusion_matrix	tn / fp / fn / tp — the raw truth behind every metric	confusion_matrix
average_precision_score	PR-AUC, the honest summary on imbalance	average_precision_score
precision_recall_curve	Precision and recall at every threshold — how you pick one	precision_recall_curve
Going deeper	scikit-learn’s guide to classification metrics	Metrics & scoring

If something looks off

• The accuracy looks great (~0.88–0.90) and you think you’re done. That’s the trap this project is built around. A model that just predicts “no” for everyone scores ~0.883 on this imbalanced data while catching zero of the actual subscribers; the real model’s 0.895 is barely above it and still finds fewer than one in five. High accuracy is the default on rare events, not an achievement — read recall and PR-AUC, not accuracy.
• The download failed / no internet. Part A needs a one-time internet connection to fetch the dataset; re-run python data/get_data.py once you’re online (it’s cached after the first success).
• A checkpoint number doesn’t match. The real dataset is fixed and the synthetic Part B data is seeded, so the numbers are stable. Re-fetch/re-generate, then re-run Part A.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Different data, a different job. A water-treatment plant wants to catch pump failures before they happen. Failures are rare (~5–6% of machine-hours). Accuracy is useless here — predict “no failure” and you’re ~94% accurate and catch nothing — so you won’t use it.

Your job is a decision, not just a diagnosis: build a classifier on data/pumps.csv, then choose a decision threshold that catches at least 80% of failures (recall ≥ 0.80) while keeping precision as high as you can. Report the honest numbers.

Same core skill as Part A (read past accuracy, use precision/recall/PR-AUC), now aimed at picking an operating point under a hard constraint.

Acceptance criteria (checkable)

Use the fixed evaluation split (train_test_split(test_size=0.25, random_state=42, stratify=y)) so your numbers are comparable. Your solution writes into part_b/out/:

• test_predictions.csv — columns y_true, y_score (your model’s probabilities on the test set).
• report.json — pr_auc, chosen_threshold, recall_at_threshold, precision_at_threshold.

Then:

python part_b/starter.py
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

You’re done when the tests pass. They require recall ≥ 0.80 at your chosen threshold with precision ≥ 0.30, a PR-AUC ≥ 0.45, and that your reported numbers match your predictions. (For reference, the solution reaches recall 0.80 at precision 0.388, PR-AUC 0.596 — you have margin.)

Constraints — what must be true, not how to get there

• Hit the recall target by choosing a threshold, not by predicting “failure” for everything (that gets recall 1.0 and precision ≈ the 5–6% base rate — the precision floor rules it out).
• Report metrics on the held-out test set, not the data you trained on.
• Your reported pr_auc / recall / precision must match what your predictions actually give (the test recomputes them).

Hints

Hint 1 — the shape of the problem

Accuracy is off the table. Train a model that outputs a probability per row, then think about the precision–recall curve rather than a single 0.5 cutoff. The noise sensors carry no signal; a model that scales its inputs will mostly ignore them.

Hint 2 — finding the threshold

precision_recall_curve(y_true, y_score) gives precision and recall at every threshold. You want the operating point where recall first reaches 0.80; among the thresholds that satisfy recall ≥ 0.80, the highest one gives you the best precision.

Hint 3 — the tradeoff (still not the code)

There’s no threshold that gives you high recall and high precision here — the data won’t allow it, and that’s the honest finding. Catching 80% of failures means accepting that most of what you flag is a false alarm. Report that precision plainly; hiding it is the mistake this project is about.

8. Self-check

You don’t need the answer key. You’re done when:

• You can state, in one sentence, why 89.5% accuracy in Part A was a bad result.
• Your Part B threshold gives recall ≥ 0.80 on the test set, and you can say the precision it costs.
• Your reported numbers are reproducible: rerun make_data.py and the split, and they don’t move (fixed seed).
• You did not reach the recall target by flagging everything — check your precision is well above the base rate.

9. Stretch

• Cost-based threshold. Instead of a fixed recall target, put a rough cost on a missed failure vs. a false alarm and pick the threshold that minimises expected cost. Show how the choice moves as the cost ratio changes.
• A stronger model. Swap the linear model for a gradient-boosted one and see how much the precision-at-recall-0.80 improves — and whether PR-AUC agrees it’s actually better.
• (Genuinely hard) Calibrate the probabilities. A model’s 0.7 often isn’t a real 70% chance. Plot a reliability curve, apply calibration, and show whether your chosen threshold should move once the probabilities mean what they say.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

• Lead with the one-liner a non-engineer gets: “89% accurate” sounded great — here’s why it was almost useless, and what I reported instead.
• Show the confusion matrix next to the accuracy, and the precision–recall curve with your chosen operating point marked.
• State the Part B tradeoff in one honest sentence: to catch 80% of failures, this is the false-alarm rate you accept. That sentence is what signals you can be trusted with a model.

Keep get_data.py, make_data.py, and the tests in the repo so anyone can reproduce your numbers from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Part A dataset	Bank Marketing (UCI Machine Learning Repository, id 222)
Publisher	S. Moro, P. Rita, P. Cortez — UCI ML Repository
Source URL	https://archive.ics.uci.edu/dataset/222/bank+marketing
License	Creative Commons Attribution 4.0 International (CC BY 4.0)
License URL	https://creativecommons.org/licenses/by/4.0/legalcode
DOI / citation	https://doi.org/10.24432/C5K306 — Moro, Rita & Cortez, A data-driven approach to predict the success of bank telemarketing
Access / verification date	2026-07-17
Redistribution	Permitted with attribution (CC BY 4.0); fetched by data/get_data.py, never committed
robots.txt checked	Yes — no robots.txt present (404); dataset repository, download is the intended use
ToS URL	https://archive.ics.uci.edu/
Part B dataset	Synthetic — generated by data/make_data.py (seed 7); no third-party rights

The bank data is real; conclusions from Part A are about this dataset. The pump data is synthetic and teaches the mechanic (threshold choice under imbalance) — it does not prove a real-world maintenance finding.

Next up

Finished this one? Continue the AI & Machine Learning track:

Pay Equity: the gap that mostly isn’t (raw vs. adjusted) →  ·  Browse all courses