Lead Scoring: a model can rank leads perfectly and still lie about the odds

Project at a glance

• Track: AI & Machine Learning
• Difficulty: Intermediate · 6/10
• Time: ~1.0 h for Part A · ~3.0 h more for the optional Part B
• You’ll use: pandas, numpy, scikit-learn, pytest
• 💻 Local setup needed (no in-browser ▶ Run yet) — a one-time, free Python install, then you just run files. Start Here walks you through it.
• New here? This is an intermediate project — it assumes you can read and run a little Python. New to the setup? Start Here walks you through it.

The data in this project is synthetic — generated locally by data/make_data.py, with no external source. That is deliberate: the whole lesson needs each lead’s true conversion probability (which no real CRM records) so calibration and profit can be measured against ground truth, and the correlated features that make the model overconfident are injected on purpose. No signup, no downloads, no GPU.

Before you start

New to calibration? Start here.

A classifier that predicts “this lead converts with probability 0.9” is making two separate claims, and they can come apart:

• Ranking — does it put likelier-to-convert leads above less-likely ones? Measured by AUC.
• Calibration — when it says “0.9”, do 90% of those leads actually convert? Measured by a reliability curve and the Brier score.

A model can rank perfectly and still be badly miscalibrated — its probabilities systematically too high or too low — because any order-preserving squish of the scores leaves the ranking (and AUC) untouched while wrecking what the numbers mean. That’s fine if you only ever sort leads. It’s a disaster the moment you multiply the probability by something — a deal’s dollar value — or compare it to a fixed threshold, because now the actual number matters.

A tiny example — two leads, a model that ranks them right but is overconfident:

lead   model says   truly converts   deal value   true expected value
A      0.95         0.70             $2,000       $1,400
B      0.80         0.55             $9,000       $4,950

Ranked by the model’s probability, A is the hotter lead. But B is worth far more in expectation. Trust the raw 0.95 and you chase A; use calibrated probabilities and you see B is the better bet. Calibration is what makes an expected-value decision correct.

What you’ll learn: measure ranking vs calibration, fix calibration without touching the ranking, and use the calibrated probabilities to decide which leads are worth pursuing. Section 3 lists exactly what you’ll be able to do.

New to Python too? See Start Here — it starts from zero, and shows how to run this in your browser with nothing to install.

1. The Brief

Your sales team can only work so many leads, and every lead worked costs real money. You’re asked to score inbound leads so reps spend time where it pays. You’ll train a scorer, discover its probabilities are badly overconfident even though it ranks leads well, calibrate it, and then use the honest probabilities to decide — lead by lead — whether the expected value justifies the cost.

Calibration is one of the most consequential and most overlooked ideas in applied ML: teams ship a model with a great AUC and then make dollar decisions off probabilities that don’t mean what they say. The fix is cheap and the payoff is direct — here, the difference between capturing 87% and 100% of the achievable profit. The skill transfers to any setting where a probability feeds a decision: credit, churn saves, fraud review, medical triage.

2. Difficulty & time

Difficulty 6 / 10. Two ideas that compound: the ranking-vs-calibration distinction (subtle, and the reason AUC can mislead), and using calibrated probabilities inside an expected-value decision under a cost. It needs scikit-learn (model, calibration, metrics) and comfort with what a probability means. Calibrated against the ledger (G10): a peer of vendor-dedupe (6) and attrition-leakage (6) — projects with a genuine methodological trap, not just a pipeline. It sits above mtouch-attribution (5) and hidden-communities (5), because the calibration insight is less intuitive than either and the payoff only appears once probability meets value; and below recommend-from-scratch (7), which builds a whole algorithm. Part B is genuinely open — you choose the model, the calibrator, and how to make the decision.

Time is separate from difficulty: Part A is about an hour; Part B is 2–3 hours. The scripts run in about a second — the time is understanding why AUC and calibration disagree.

3. What you’ll be able to do after

• Separate ranking from calibration — measure AUC and a reliability curve, and explain why a model can ace one and fail the other.
• Read a reliability curve and the Brier score — and diagnose over- vs under-confidence.
• Calibrate a classifier (isotonic / Platt) and show the ranking is unchanged while the probabilities become trustworthy.
• Make an expected-value decision — combine calibrated probability with value and cost to decide, and measure the profit it captures.

The finished result

By the end of Part A you’ll have caught a model that ranks leads well yet lies about the odds — and fixed it without disturbing the ranking:

AUC (ranking):        0.81 raw  ->  0.81 calibrated   (unchanged)
Brier (calibration):  0.25 raw  ->  0.18 calibrated   (lower is better)
"hot leads" (p >= 0.80):  raw flags 1,430 that convert at only 0.76 (over-promised);
                          calibrated flags 702 that genuinely convert at 0.88

Same ranking, honest probabilities — and in Part B those calibrated numbers turn into money, capturing ~1.00 of the achievable profit where the raw ones capture only ~0.87.

4. Prereqs & time box

You can write a function, use a pandas DataFrame, and fit a scikit-learn classifier (.fit/.predict_proba). You’ve seen AUC before. No prior calibration experience needed — Part A builds it. The models are tiny and CPU-only.

Where to write and run code

Two ways to run this project — pick either:

• In your browser (no install). If the project page shows a ▶ Run button, click it to run Part A right here — nothing to set up. Part A uses pandas, numpy, and scikit-learn, which run in the browser.
• On your computer. The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements, and running Part A step by step — is covered once in Start Here. Come back here once python -c "import sklearn" runs without error. Part B (writing your own solution) needs this local setup.

Time box

• Part A — Guided Build: 60–90 minutes. Hard cap.
• Part B — Your Turn: 2–4 hours. Hard cap.

5. Setup & data

The data is synthetic and generated locally — see the banner above and SOURCES.md for full provenance. One command creates both parts’ data:

python data/make_data.py

That writes four CSVs into data/ (git-ignored; regenerate anytime, byte-for-byte identical because the seed is fixed):

• Part A — train_a.csv (6,000 leads) and eval_a.csv (3,000). Columns x0..x11 (twelve correlated signals) and y (converted 0/1).
• Part B — train_b.csv (6,000, with y) and leads_b.csv (4,000 to score: lead_id, x0..x11, value — no y; the grader holds the outcomes).

The twelve features are all noisy views of one latent “lead quality,” which is what makes a Naive Bayes model overconfident — the setup the project is built around.

6. Part A — Guided Build

You’ll train a scorer and interrogate its probabilities. Run the whole thing, or (better) one # %% cell at a time — see Start Here:

python part_a/calibrate.py

Prefer the browser? If this page has a ▶ Run button, click it instead of the command above — Part A runs in your browser with nothing to install. The STEP output and every checkpoint below are identical either way.

The full, commented script is in part_a/calibrate.py. The spine:

Step 1 — Train and check the ranking. Fit GaussianNB; measure AUC.

Checkpoint: AUC ≈ 0.81 — it orders leads well.

Step 2 — Check the probabilities. Reliability curve + Brier.

Checkpoint: badly overconfident — where it predicts ~0.99 it converts ~0.76; where it predicts ~0.01 it converts ~0.28. Brier ≈ 0.25.

Step 3 — Calibrate (isotonic).

Checkpoint: reliability points fall onto the diagonal; Brier drops to ≈ 0.18; AUC is unchanged at 0.81 — calibration re-maps the scores monotonically, so it never reorders leads.

Step 4 — Act on the probabilities. A “hot lead” desk that pursues p ≥ 0.80.

Checkpoint: the raw model’s “≥80%” list (~1,430 leads) actually converts at 0.76 — it over-promised — and real buyers hide in the bucket it calls “<2%” (they convert ~0.24). The calibrated “≥80%” list converts at 0.88. Writes part_a/reliability_a.csv.

Why this and not that

• Why AUC isn’t enough. AUC only sees order. Two models with identical rankings — one honest, one wildly overconfident — score the same AUC. The instant you attach a probability to a dollar amount or a threshold, order stops being enough.
• Why Naive Bayes is overconfident here. It assumes features are independent and multiplies their evidence. When a dozen features are really one signal in disguise, it multiplies the same evidence twelve times and slams the probability to 0 or 1. (Great teaching case; also a real gotcha with correlated features.)
• Why calibration doesn’t change AUC. Isotonic and Platt scaling are monotonic — they squeeze and stretch the scores but never swap their order. So ranking (AUC) is invariant while the numbers become trustworthy. That invariance is the lesson.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
GaussianNB().fit(X, y)	Train the (deliberately overconfident) base scorer	GaussianNB
roc_auc_score(y, p)	Measure ranking quality	roc_auc_score
calibration_curve(y, p, n_bins)	Predicted probability vs actual frequency	calibration_curve
brier_score_loss(y, p)	One number for calibration quality	brier_score_loss
CalibratedClassifierCV(est, method="isotonic")	Learn a calibration map by cross-validation	CalibratedClassifierCV

If something looks off

• After calibrating, the AUC printed the same number as before (0.81 both times) — did calibration do nothing? It fixed exactly what was broken: the probabilities (the Brier score drops from 0.25 to 0.18), while leaving the ranking untouched. AUC scores only ranking, and calibration re-maps the scores monotonically, so it can never reorder leads — that unchanged number is the whole point, not a bug. A high AUC is never proof the probabilities are trustworthy; that is what the reliability curve and the Brier score are for.
• A checkpoint number doesn’t match. The data is seeded, so the numbers are exact. Re-run python data/make_data.py (you may have skipped or edited it), then re-run Part A.
• Setup problems (Python not found, pip, FileNotFoundError, re-activating the venv): see the troubleshooting list in Start Here.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Same core skill, now spending money. Each of the 4,000 leads in leads_b.csv has a dollar value, and working a lead costs a flat $4,000. Decide which leads to pursue: pursue a lead only if its expected value — P(convert) × value — beats the cost. The starter (part_b/starter.py) makes exactly this decision with the raw Naive Bayes probabilities and captures only ~0.87 of the achievable profit — below the floor.

The transfer: in Part A you saw the probabilities were overconfident. Now that error costs money — an inflated probability makes a losing lead look profitable (you pursue and burn the $4,000) and a deflated one makes a real buyer look hopeless (you skip it). Calibrate first, then decide.

Acceptance criteria (checkable)

Your solution writes into part_b/out/:

• decisions.csv — columns lead_id, pursue (1 = pursue, 0 = skip), one row per lead.
• report.json — with keys leads, pursued, achieved_metric (you can leave achieved_metric null; the grader scores you).

Then:

python part_b/starter.py
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

You’re done when the tests pass: your pursued set captures ≥ 0.92 of the oracle’s net profit — sum(true_p × value − cost) over the leads you pursue, divided by the same for the decision made with the true probabilities (the reference reaches ~1.00). The raw-probability decision captures ~0.87 and fails.

Constraints — what must be true of your solution, not how to get there

• Pursue a lead only when your estimated expected value exceeds the $4,000 cost — not by rank, not by probability alone.
• Calibrate the probabilities before the decision; raw Naive Bayes probabilities are overconfident.
• Fit only on train_b.csv; you don’t have outcomes for the scoring leads.
• Decide on every lead (one row per lead_id).

Hints

Hint 1 — why raw probabilities lose money

Compare, on a few leads, raw p × value against calibrated p × value. Where raw p is inflated toward 1, p × value overstates the expected value and the lead clears the cost bar when it shouldn’t.

Hint 2 — the calibrated model

CalibratedClassifierCV(GaussianNB(), method="isotonic", cv=5).fit(train[FEATS], train.y) gives you calibrated predict_proba. It’s the same tool from Part A, now feeding a decision.

Hint 3 — the decision (still not the code)

For each lead compute p_calibrated * value; pursue it iff that exceeds 4000. Write pursue = 1 for those, 0 for the rest.

8. Self-check

You don’t need the answer key. You’re done when all of these hold:

• pytest reports captured ratio ≥ 0.92 (aim for ~1.00 — calibrated expected values nearly match the oracle’s).
• Swapping the calibrated model for the raw one drops your captured profit (you can see calibration is doing the work, not luck).
• Re-running from python data/make_data.py reproduces the same numbers — the data is seeded.

And the tell-tale: your pursued count should land near the oracle’s (~1,450), not far above it (over-pursuing on inflated probabilities) or far below (skipping real buyers on deflated ones).

9. Stretch

• Platt vs isotonic. Calibrate with method="sigmoid" (Platt) instead and compare Brier and captured profit. Decide which you’d trust with 6,000 training rows and why.
• Calibrate a different model. Swap in a small DecisionTreeClassifier or LogisticRegression, check whether it even needs calibration (logistic regression usually doesn’t), and explain the difference.
• (Genuinely hard) Capacity, not just cost. Add a hard limit — reps can only work 1,000 leads — and choose the set that maximises expected profit under that budget. Now ranking by expected value matters and the calibration has to be right; show both effects.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

• State the finding in one sentence a sales leader gets: “the model’s AUC was fine, but its probabilities were overconfident — calibrating them lifted captured profit from 87% to 100% of the achievable.”
• Show the reliability curve before and after calibration, with AUC annotated as unchanged.
• Include the pursue decision and one paragraph on why a good AUC is not permission to trust the probabilities. That paragraph is the part that signals you understand calibration, not just .fit().

Keep make_data.py and the tests in the repo so anyone can reproduce your numbers from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Dataset	Synthetic inbound sales leads: 12 correlated signals + conversion + deal value (Part A 6,000/3,000; Part B 6,000/4,000)
Type	Synthetic
Generated by	data/make_data.py (this repo), seeds 20260726/20260727 (A), 20260728/20260729 (B)
Source URL	Not applicable — generated, not fetched
License	Not applicable — synthetic, no third-party rights
Access / generation date	2026-07-18
Redistribution	Permitted — synthetic; output regenerated from the seeded script, never committed
robots.txt checked	Not applicable — no scraping
ToS URL	Not applicable — no external source

Synthetic data teaches the mechanic; it does not prove a finding about the real world. The overconfidence was engineered on purpose (correlated features + Naive Bayes) — recovering and fixing it demonstrates the technique, not a claim about any real lead model.

Next up

Finished this one? Continue the AI & Machine Learning track:

Recommendations from scratch: matrix factorization that beats the popularity baseline →  ·  Browse all courses