Attrition Leakage: the model that's too good to be true

Project at a glance

• Track: AI & Machine Learning
• Difficulty: Intermediate · 6/10
• Time: ~1.25 h for Part A · ~3.5 h more for the optional Part B
• You’ll use: pandas, numpy, scikit-learn, pytest
• 💻 Local setup needed (no in-browser ▶ Run yet) — a one-time, free Python install, then you just run files. Start Here walks you through it.
• New here? This is an intermediate project — it assumes you can read and run a little Python. New to the setup? Start Here walks you through it.

The data in this project is synthetic — generated locally by data/make_data.py, with no external source. That is deliberate: to teach leakage you need to know exactly which features leak and how strongly, which only a known generative model gives you. No signup, no downloads, no GPU.

Before you start

New to target leakage? Start here — it’s the trap this whole project is about.

Target leakage is when a feature in your training data secretly encodes the answer — usually because it’s recorded at the same time as, or after, the outcome. Your model looks brilliant in development and then falls apart in production, because at prediction time that feature isn’t available yet (or doesn’t mean what it did in training).

The classic example is exactly this project: predicting which employees will leave. If your data includes exit_interview_done or equipment_returned, your model will be nearly perfect — but only because those things happen because someone left. When you try to predict who will leave next quarter, nobody has done an exit interview yet, so the feature is blank and the model is useless.

The tell is a score that’s too good: real attrition is not perfectly predictable, so an AUC of 0.99 is a bug report, not a triumph. The skill is to detect the leaks — a feature that alone predicts the target almost perfectly is the red flag — and remove them before you trust anything.

What you’ll learn: spot a suspiciously perfect model, audit each feature’s individual predictive power to find the leaks, drop them, and report the honest score. Section 3 lists exactly what you’ll be able to do.

New to Python too? See Start Here — it starts from zero, and shows how to run this in your browser with nothing to install.

1. The Brief

You’re handed an HR dataset and asked to build a model that predicts employee attrition. The first model you train scores an AUC near 1.00. Instead of celebrating, you’ll do the thing that separates a practitioner from a beginner: distrust it, find the leakage, and report the honest number the business can actually act on.

Leakage is the single most common reason a model that “worked in the notebook” fails in production — and it’s insidious because every metric looks great right up until deployment. Interviewers probe for it; postmortems keep rediscovering it. Learning to smell a too-good score and hunt down the leak is a core, transferable ML skill, not an HR-specific trick.

2. Difficulty & time

Difficulty 6 / 10. It needs a model, a per-feature diagnostic (single-feature AUC), and the judgment to distinguish a leak from a merely-strong honest feature — plus the discipline to lower a great-looking score. Calibrated against the ledger (G10): a peer of vendor-dedupe (6), which likewise composes several diagnostic steps and a threshold decision. It sits above pay-equity-gap and accuracy-is-lying (both 5), which each run one analysis with a conceptual trap, and below recommend-from-scratch (7), which derives an algorithm from scratch. The modelling here is a few lines of scikit-learn; the difficulty is the detective work and the judgment.

Time is separate from difficulty: Part A is about 75 minutes; Part B is 2–4 hours of your own work.

3. What you’ll be able to do after

• Recognize a leaked model — treat a near-perfect score as a symptom to investigate, not a result to ship.
• Audit features for leakage — rank each feature by its individual AUC against the target and read an implausibly high value as a red flag.
• Tell a leak from an honest strong feature — a leak scores far above the clean pack, even when it’s not literally 1.0.
• Report an honest baseline — retrain without the leaks and quote the number that will survive production.

The finished result

By the end of Part A you’ll have caught a model that scores almost perfectly, traced the five features leaking the answer, and reported the honest number the business can actually act on:

AUC with all 12 features:        1.000   <-- perfect, and that is the warning sign
single-feature audit:  5 leaks at 0.87-0.97 (exit_interview_done, ...); clean pack tops out at 0.784
AUC with the 5 leaks removed:    0.908   <-- lower, and the number you can trust

The near-perfect score was the bug, not the win: drop the leaked features and the honest model lands at 0.908 — well short of perfect, which is exactly what real attrition looks like.

4. Prereqs & time box

You can write a function and use a pandas DataFrame; you’ve seen a classifier and the idea of AUC once (or will treat roc_auc_score as “1.0 = perfect ranking, 0.5 = coin flip”). No deep ML background needed.

Where to write and run code

Two ways to run this project — pick either:

• In your browser (no install). If the project page shows a ▶ Run button, click it to run Part A right here — nothing to set up. Part A uses pandas, numpy, and scikit-learn, which run in the browser.
• On your computer. The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements, and running Part A step by step — is covered once in Start Here. Come back here once python -c "import sklearn" runs without error. Part B (writing your own solution) needs this local setup.

Time box

• Part A — Guided Build: 60–90 minutes. Hard cap.
• Part B — Your Turn: 2–4 hours. Hard cap.

5. Setup & data

The data is synthetic and generated locally — see the banner above and SOURCES.md for full provenance. One command creates both files:

python data/make_data.py

That writes two CSVs into data/ (git-ignored; regenerate anytime, byte-for-byte identical because the seed is fixed):

• Part A — employees_a.csv (1,500 rows, 12 features).
• Part B — employees_b.csv (4,000 rows, 12 features — with different leaky columns).

Each row has clean features (satisfaction, overtime, commute, tenure, …), some leaky features, and the target left (1 = the employee left). Which features are leaky is not labelled — you detect it.

6. Part A — Guided Build

You’ll work employees_a.csv. Run the whole thing, or (better) one # %% cell at a time — see Start Here:

python part_a/detect_leakage.py

Prefer the browser? If this page has a ▶ Run button, click it instead of the command above — Part A runs in your browser with nothing to install. The STEP output and every checkpoint below are identical either way.

The full, commented script is in part_a/detect_leakage.py. The spine:

Step 1 — Train on all features. Fit a logistic model, measure AUC on a held-out split.

Checkpoint: AUC ≈ 1.00. That should worry you — real attrition isn’t perfectly predictable, so a near-perfect score usually means a feature is leaking the answer.

Step 2 — Audit each feature alone. Compute every feature’s single-feature AUC against the target (max(auc, 1-auc), since a feature can leak by being high or low for leavers).

Checkpoint: clean features top out around 0.78; five features score 0.87–0.97 (exit_interview_done, equipment_returned, access_revoked, offboarding_tickets, final_survey_sent). Those happen because someone left.

Step 3 — Separate leaky from clean. Flag features whose single-feature AUC is clearly above the clean pack (here, > 0.83).

Checkpoint: 5 leaky, 7 clean. The leaky ones are recorded at/after exit — blank at prediction time.

Step 4 — Retrain on clean features. The honest model.

Checkpoint: AUC falls from ≈1.00 to ≈ 0.91 — lower, but the number you can trust.

Writes part_a/feature_audit_a.csv.

Why this and not that

• Why distrust a near-perfect score instead of shipping it. Genuine outcomes have irreducible noise. An AUC that rounds to 1.00 almost always means the model can see the answer — the correct reaction is suspicion, not a victory lap.
• Why single-feature AUC, not just model feature-importance. Importance tells you what the model used; single-feature AUC tells you what each feature could do on its own, which is the direct leakage signal — and it’s model-agnostic.
• Why a leak isn’t “just a strong feature.” Both help the model. The difference is availability at prediction time: a leak’s value is a consequence of the outcome, so it won’t exist (or won’t mean the same thing) when you predict. That’s a reasoning step, not a threshold — the AUC just tells you where to look.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
roc_auc_score(y, x)	Score a feature’s (or model’s) ranking of the target	roc_auc_score
LogisticRegression().fit	Train the classifier	LogisticRegression
train_test_split(...)	Hold out data so the AUC isn’t measured on training rows	train_test_split
predict_proba(X)[:, 1]	Get the probability used for AUC	predict_proba
df.columns / list-comp	Select feature columns (exclude id and target)	pandas

If something looks off

• Step 1’s AUC is ~1.00 and it feels like a win — isn’t a near-perfect model the goal? That is the bug, not the prize. Real attrition is not perfectly predictable, so a score that rounds to 1.00 is the tell-tale of target leakage: some feature is quietly encoding the answer. The suspiciously high number is the symptom you are here to chase, not a result to ship — Steps 2–4 hunt down the leaks and report the honest ~0.91.
• A checkpoint number doesn’t match. The data is seeded, so the numbers are exact. Re-run python data/make_data.py (you may have skipped or edited it), then re-run Part A.
• Setup problems (Python not found, pip, FileNotFoundError, re-activating the venv): see the troubleshooting list in Start Here.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Same core skill, more rows, and — crucially — the leaky features have different names. You can’t recognize them from Part A; you have to detect them. Audit employees_b.csv, decide which features are leaky, and output your exclude-set. The starter (part_b/starter.py) flags only the blatant leaks (single-feature AUC > 0.95) and scores ~0.75, below the floor. Its # TODO is the same per-feature AUC audit you watched in Part A, STEP 2 — applied to the renamed columns, with the threshold lowered — so you’re extending code you’ve already seen run, not writing it from scratch.

The transfer: two of Part B’s leaks are subtler (single-feature AUC ~0.87, not ~0.97). A threshold tuned to “almost 1.0” walks right past them. The clean features here top out near 0.79, so a leak that scores 0.87 is still a screaming outlier — you just have to compare each feature to the clean pack, not to 1.0.

Acceptance criteria (checkable)

Your solution writes into part_b/out/:

• flagged.csv — one row per feature: feature, is_leaky (1 if you’d exclude it).
• report.json — with keys features, flagged_leaky, achieved_metric (you can leave achieved_metric null; the grader scores you).

Then:

python part_b/starter.py
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

You’re done when the tests pass: your exclude-set reaches F1 ≥ 0.85 against the true leaky set (the reference reaches 1.00). Catching only the blatant leaks scores ~0.75 and fails.

Constraints — what must be true of your solution, not how to get there

• Detect the leaks from the data — don’t hard-code Part A’s leaky names (they don’t exist here).
• Judge each feature against the clean pack, not against 1.0 — the subtle leaks sit well above the honest features but below “almost perfect.”
• Emit a verdict for every feature.

Hints

Hint 1 — start from the audit

Compute every feature’s single-feature AUC (max(auc, 1-auc)) and sort. You’ll see an obvious gap between the clean features and the leaks.

Hint 2 — where to put the line

The clean features cluster below ~0.79. Set your leak threshold just above that pack (≈0.83), not up at 0.95 — otherwise the two subtle leaks slip through.

Hint 3 — sanity-check (still not the code)

You should flag 5 features. If you flag 3, your threshold is too high (you kept two leaks); if you flag 7+, it’s too low (you dropped an honest feature).

8. Self-check

You don’t need the answer key. You’re done when all of these hold:

• pytest reports F1 ≥ 0.85 (aim for 1.00 — the gap between clean and leaky is clear once you compare to the clean pack).
• You flagged ~5 features, not 3 (missed subtle leaks) or 7+ (dropped honest ones).
• Re-running from python data/make_data.py reproduces the same numbers — the data is seeded.

And the tell-tale: if your “honest” model (leaks removed) still scores ~1.00, you missed a leak; a real attrition model lands well short of perfect.

9. Stretch

• Prove the failure. Simulate production: zero out the leaky columns at test time and show the all-features model’s AUC collapses toward the honest model’s — the leak was carrying it.
• Beyond single-feature AUC. Some leaks only reveal themselves in combination. Check whether any pair of “clean” features jointly predicts the target far better than either alone.
• (Genuinely hard) Temporal leakage. Add a timestamp to each feature and flag any feature recorded after the attrition date — leakage detection by provenance, not just by AUC.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

• State the lesson in one sentence a hiring manager gets: “my first model hit AUC 1.00 — here’s how I found the leakage and reported the honest 0.91.”
• Show the audit table (single-feature AUC per feature) with the leaks highlighted, and the before/after model AUC.
• Include one paragraph on why those features leak (recorded at/after exit) — the reasoning, not just the threshold. That paragraph is what signals you understand leakage.

Keep make_data.py and the tests in the repo so anyone can reproduce your numbers from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Dataset	Synthetic HR attrition with injected target leakage (Part A 1,500, Part B 4,000 employees)
Type	Synthetic
Generated by	data/make_data.py (this repo), seeds 20260718 (A) and 20260719 (B)
Source URL	Not applicable — generated, not fetched
License	Not applicable — synthetic, no third-party rights
Access / generation date	2026-07-18
Redistribution	Permitted — synthetic; output regenerated from the seeded script, never committed
robots.txt checked	Not applicable — no scraping
ToS URL	Not applicable — no external source

Synthetic data teaches the mechanic; it does not prove a finding about the real world. The leaks were injected on purpose — detecting them demonstrates the technique, not a claim about any real workforce.

Next up

Finished this one? Continue the AI & Machine Learning track:

Lead Scoring: a model can rank leads perfectly and still lie about the odds →  ·  Browse all courses