Feature scaling: why one big-numbered column can wreck a distance-based model

Data: Part A uses a real dataset — scikit-learn’s Wine Recognition data (bundled, no download). Part B uses a synthetic clustering problem generated locally by data/make_data.py (seeded, reproducible). Free, no signup, no GPU. Synthetic Part B is deliberate: to score whether scaling recovered the true groups you must know them, which only generated data gives you.

Before you start

New to this? Two minutes of vocabulary, because the whole project turns on it.

Many models measure how close two rows are — k-nearest-neighbours asks “which training points are nearest?”, k-means groups points by distance to a centre. Distance is usually plain Euclidean: square the difference in each feature, add them up, square root.

The trap: that sum is dominated by whichever feature has the biggest numbers. If one column runs 0–1500 and another runs 0–1, the second column contributes almost nothing to the distance — the model effectively ignores it.
Standardizing (z-scoring): rescale each feature to mean 0, standard deviation 1, so every feature contributes on equal footing. (StandardScaler does this.)
Scale-sensitive vs. not: distance- and gradient-based models (kNN, k-means, SVM, PCA, regularized regression) care a lot about scale. Tree-based models (decision trees, random forests) don’t — they split one feature at a time and never compare magnitudes across features.

A tiny example — rank wines by “distance” using raw numbers and you’re really ranking them by proline (which runs into the hundreds) while ignoring twelve other measurements. Put every feature on the same scale first and all thirteen count.

What you’ll learn: to spot when feature scales will sabotage a model, to standardize correctly (including without leaking test information), and to know which models need it and which don’t.

1. The Brief

You’ll run k-nearest-neighbours on a real dataset of wine chemistry and watch it score a mediocre 0.68 — not because kNN is weak, but because one column (proline) has such big numbers it drowns out the other twelve in the distance calculation. Standardize the features first and the same model on the same data jumps to 0.97. Then, in Part B, you’ll meet a clustering problem rigged with two big irrelevant columns that hijack the groups, and you’ll recover the true structure — going from 0.36 (raw) to 1.00 (scaled) — by scaling before you cluster.

This is one of the most common silent failures in applied machine learning: a model underperforms and everyone blames the model, when the real culprit is that nobody put the features on a common scale. It’s cheap to fix and easy to get subtly wrong (scaling after you split leaks; scaling a tree model wastes effort; forgetting to scale a distance model quietly halves your accuracy). Knowing when scale matters — and when it doesn’t — is basic literacy for anyone who trains models.

2. Difficulty & time

Difficulty 4 / 10. It sits just above the difficulty-3 builds (#silent-row-loss, #idempotent-loads, #benford-fraud): those turn on one mechanic, whereas this one has you compare two models across two scalings, reason about which algorithms are scale-sensitive, handle scaling without leakage inside cross-validation, and then transfer the idea from supervised classification to unsupervised clustering. It stays below the difficulty-5 group (#reorder-point-trap, #accuracy-is-lying, #hidden-communities) because the central idea — put features on a common scale before a distance-based model — is a single concept, and Part A’s path is fully specified. Anchored to the rubric’s 3–4 band and those named ledger projects (G10).

Time is separate from difficulty: the scripts run in about a second (measured Part A wall-clock 1.27s). Part A is about 45 minutes of reading and running; Part B is 2 hours of your own work.

3. What you’ll be able to do after

Diagnose when differing feature scales will sabotage a distance- or gradient-based model — by looking at the feature ranges, not by guessing.
Standardize features with StandardScaler, and do it inside a pipeline so the scaler is fit on training data only (no leakage).
Name which model families need scaling (distance/gradient) and which don’t (trees), and justify why.
Transfer the fix to a new setting: recover clusters that raw-scale distance would hide.

The finished result

By the end of Part A you’ll have run the same kNN on the same wine data twice — once on the raw features and once standardized — and watched scaling alone close the gap:

widest feature `proline` ranges 1402  vs  narrowest ~0.53   ->  a 2645x scale gap
kNN accuracy, raw features:   0.680
kNN accuracy, standardized:   0.972   (+0.29 — same model, same data, only the scale changed)

One big-numbered column was drowning out twelve others; put every feature on a common scale and kNN can use all thirteen. In Part B the same fix takes a rigged clustering problem from 0.36 (raw) to 1.00 (scaled).

4. Prereqs & time box

You can write a function and use a pandas DataFrame, and you’ve trained at least one model (you know what train/test and accuracy mean). No maths beyond “Euclidean distance adds up squared differences,” which is explained above.

Where to write and run code

Two ways to run this project — pick either:

In your browser (no install). If the project page shows a ▶ Run button, click it to run Part A right here — nothing to set up. Part A uses pandas, numpy, and scikit-learn, which run in the browser.
On your computer. The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements, and running Part A step by step — is covered once in Start Here. Come back here once python -c "import sklearn" runs without error. Part B (writing your own solution) needs this local setup.

Time box

Part A — Guided Build: 30–60 minutes. Hard cap 90.
Part B — Your Turn: 1.5–2.5 hours. Hard cap 4.

5. Setup & data

Part A (real). One command reads the Wine Recognition dataset from scikit-learn and writes it locally (git-ignored, never committed):

python data/get_data.py

Part B (synthetic). One command generates the clustering problem (seeded, byte-for-byte reproducible):

python data/make_data.py

That writes data/cluster_features.csv — sensor rows with three real operating modes hidden in three well-behaved columns, plus two big-numbered irrelevant columns. The true modes are documented in data/make_data.py and used by the grader; you won’t see them.

6. Part A — Guided Build

Run the full script and read along:

python data/get_data.py         # once
python part_a/scale_knn.py

Prefer the browser? If this page has a ▶ Run button, click it instead of the command above — Part A runs in your browser with nothing to install. The STEP output and every checkpoint below are identical either way.

The full, commented script is in part_a/scale_knn.py.

Step 1 — Look at the feature scales.

Checkpoint: the widest feature (proline) ranges over ~1400 while the narrowest ranges under 1 — a ratio in the thousands. In plain Euclidean distance, proline is almost the only feature that counts.

Step 2 — kNN on the raw features.

Checkpoint: accuracy ≈ 0.68. Distance is dominated by one column, so the other twelve measurements barely affect which neighbours are “near.”

Step 3 — Standardize (inside the cross-validation, so the scaler sees only training folds), then kNN.

Checkpoint: accuracy ≈ 0.97 — a jump of ~0.29 from the same model on the same data. Once every feature contributes equally, kNN can use all thirteen.

Step 4 — The rule: scale-sensitive models (distance/gradient) need standardizing; tree models don’t; and fit the scaler on training data only.

Why this and not that

Why standardize (z-score) rather than min–max scale? Both put features on a comparable footing; standardizing centres on the mean and divides by the standard deviation, so it’s robust to the range being set by a couple of extreme values. Min–max (0–1) is fine too, but a single outlier can squash everything else into a sliver. For distance models, either beats not scaling; the mistake is doing neither.
Why scale inside the cross-validation, not once up front? Fitting the scaler on the whole dataset lets the test folds influence the mean and standard deviation used on the training folds — a small but real leak that flatters your score. Putting the scaler in a Pipeline means each fold re-fits it on training data alone.
Why not just scale everything, always? Tree-based models (random forests, gradient boosting) are scale-insensitive — they split one feature at a time and never compare magnitudes — so scaling neither helps nor hurts them, and skipping it keeps the features interpretable. Knowing which models care is the point.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
`StandardScaler`	Rescale each feature to mean 0, std 1	StandardScaler
`KNeighborsClassifier`	The distance-based classifier that scale makes or breaks	KNeighborsClassifier
`Pipeline`	Bundle scaler + model so scaling happens inside each CV fold	Pipeline
`cross_val_score`	Honest accuracy across folds	cross_val_score
`KMeans`	(Part B) the distance-based clusterer	KMeans

If something looks off

I scaled the whole dataset once, up front, and the accuracy came out a touch higher — isn’t that better? No — that higher number is the leak. Fitting StandardScaler on every row lets the held-out test folds nudge the mean and standard deviation used on the training folds, quietly flattering the score. The Pipeline in Step 3 re-fits the scaler on each fold’s training rows only, so trust that number, not the up-front one — this is exactly the leakage the project warns about.
A checkpoint number doesn’t match. The data is seeded (or ships inside scikit-learn), so the numbers are exact. Re-run the data step, then re-run Part A.
Setup problems (Python not found, pip, FileNotFoundError, re-activating the venv): see the troubleshooting list in Start Here.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Different setting, same trap. data/cluster_features.csv is a delivery-fleet sensor log: three genuine operating modes are hidden in three well-behaved columns (f_a, f_b, f_c), but the log also has two big-numbered columns — packet_counter and uptime_s — that carry no mode information and dominate raw distance. There are no labels: this is clustering.

Your job: cluster the rows into the 3 operating modes as well as you can. Same core skill as Part A (put features on a common scale before a distance-based method), now for unsupervised clustering where the scale trap is even more punishing.

Acceptance criteria (checkable)

Fill in part_b/starter.py, then:

python part_b/starter.py            # writes part_b/out/
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

Your solution writes into part_b/out/:

assignments.csv — row, cluster (a cluster id for every row).
report.json — rows, clusters, cluster_accuracy, achieved_metric.

You’re done when the tests pass. The grader regenerates the true modes from the seed (you never see them) and scores your clustering by accuracy (each cluster mapped to its majority true mode). It requires a cluster for every row and accuracy ≥ 0.75. The reference reaches 1.00; clustering the raw features scores about 0.36 (barely better than guessing among three groups).

Constraints — what must be true, not how to get there

Recover the modes from the features, into exactly 3 clusters.
Handle the two big-scale irrelevant columns so they don’t dominate the distance — the whole difference between 0.36 and 1.00 is here.
Your reported cluster_accuracy must match your assignments (the grader recomputes it).

Hints

Hint 1 — why raw clustering fails

Look at the column ranges. packet_counter and uptime_s are orders of magnitude larger than f_a, f_b, f_c. In Euclidean distance they swamp everything, so k-means groups rows by counter/uptime — which have nothing to do with the modes.

Hint 2 — the fix

Standardize the features before k-means (the same StandardScaler from Part A). Then every column, including the three that actually carry the modes, contributes equally.

Hint 3 — a subtlety (still not the code)

Scaling makes the irrelevant columns stop dominating, but they’re still noise. Here the three real features separate the modes cleanly enough that standardizing is all you need. In a harder version you’d also drop or down-weight columns you know are irrelevant — scaling equalises influence, it doesn’t remove useless features.

8. Self-check

You don’t need the answer key. You’re done when:

You can explain, in one sentence, why raw-feature kNN scored 0.68 on the wine data.
Your Part B clustering accuracy is far above the ~0.36 raw-features level, and you can point to the two columns that caused the trap.
You can name two model types that need scaling and one that doesn’t.
You scaled without leakage (fit on training data only) in Part A’s cross-validation.

9. Stretch

Min–max vs. standardize vs. robust. Try MinMaxScaler and RobustScaler on the wine data and compare kNN accuracy. Add an extreme outlier to one feature and see which scaler copes.
Prove trees don’t care. Run a random forest on the raw and scaled wine data and confirm the accuracy barely moves — then explain why, from how a tree splits.
(Genuinely hard) Scaling isn’t feature selection. Extend Part B so the irrelevant columns are not just large but numerous. Show that standardizing alone stops helping once noise features outnumber signal, and that you now need to select features, not just rescale them.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

Lead with the one-liner anyone gets: “The same kNN scored 0.68 on raw wine data and 0.97 after scaling — one big-numbered column was drowning out twelve others.”
Show the feature-range bar chart (proline towering over the rest) and the before/after accuracy for both Part A (classification) and Part B (clustering).
State the rule in one sentence: standardize before any distance- or gradient-based model, fit the scaler on training data only, and don’t bother for trees.

Keep get_data.py, make_data.py, and the tests in the repo so anyone can reproduce your numbers from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Part A dataset	Wine Recognition (real), bundled in scikit-learn
Origin	UCI ML Repository (Forina et al.); via `sklearn.datasets.load_wine`
License	Freely available for research/education; shipped in scikit-learn (BSD-3-Clause)
Part B dataset	Synthetic — clustering with a scale trap, `data/make_data.py` (seed 20260718)
Access / verification date	2026-07-18
Redistribution	Wine fetched from the installed package, never committed; synthetic Part B never committed

The wine results are about that real dataset; the Part B numbers are about a synthetic problem and teach the mechanic (scale before distance-based methods), not a real-world finding.

Next up

Finished this one? Continue the AI & Machine Learning track:

Is this review positive? Reading sentiment from text → · Browse all courses