Recommendations from scratch: matrix factorization that beats the popularity baseline

Data: both parts use synthetic data, generated locally by data/make_data.py (seeded, reproducible). Nothing is downloaded; no signup, no GPU (a few hundred users and items train in about a second). Synthetic is the right call: the ratings are built from hidden “taste” factors we control, so we can check whether your model recovered real structure or just memorised noise.

Before you start

New to this? A few minutes of vocabulary — this one assumes you’ve trained a model before, but here’s the shape of the problem.

You have a big table of users × items, and almost all of it is blank: each user has rated only a handful of movies. You want to fill in the blanks — predict how a user would rate something they haven’t seen — so you can recommend the ones they’d love.

Latent factors — the trick behind matrix factorization. We assume each user has a few hidden “taste” numbers (how much they like, say, action vs. romance) and each movie has matching numbers (how action-y vs. romance-y it is). A rating is roughly the dot product of the two: taste lined up with content. We never label these dimensions; we learn them from the ratings we have.
Matrix factorization — finding those user-factor and item-factor tables so their product reproduces the known ratings, then using them to predict the unknown ones.
Gradient descent — how we learn them: start random, repeatedly nudge every factor in the direction that lowers the error on the ratings we can see.
RMSE vs. precision@k — two ways to grade. RMSE measures how close predicted ratings are to true ones (Part A). Precision@k measures ranking: of the top k items you recommend, how many the user actually loves (Part B). A recommender’s job is usually the second one.

A tiny example — if a user’s taste vector is “loves action, hates romance” and a movie’s is “very action, no romance,” their dot product is high, so the model predicts a high rating even if that exact pair was never in the data. That generalisation is what an average-rating baseline can never do.

1. The Brief

You’ll build matrix factorization from nothing but numpy — learn the hidden taste factors by gradient descent — and predict held-out movie ratings at an RMSE of 0.56, less than half the error of the obvious “just predict the user’s average” baseline (1.37). Then, in Part B, you’ll point the same idea at a different problem: ranking a bookshop’s catalogue for each reader and recommending the top five, hitting precision@5 of 0.90 where ranking by raw popularity manages only 0.33.

This is the algorithm that won the Netflix Prize and still underpins recommendation at most companies that do it. It’s here at the top of the difficulty range because it asks you to implement a learning algorithm, not call one: you write the update rule, you regularise it, you watch it converge, and you learn the two things that separate a real recommender from a demo — that predicting a rating and ranking a shortlist are different jobs with different scores, and that beating a popularity baseline is the only result that means anything.

2. Difficulty & time

Difficulty 7 / 10. This is the batch’s hardest project so far, clearly above the difficulty-5 group (#reorder-point-trap, #accuracy-is-lying, #hidden-communities): those use a library algorithm and reason about it, whereas this one has you implement a learning algorithm from scratch — the gradient-descent update for biased matrix factorization, with regularisation — and then transfer it to a different objective. It is well above the difficulty-3 builds (#silent-row-loss, #idempotent-loads, #benford-fraud), which turn on a single mechanic. It assumes you’re comfortable with arrays and have trained a model before. It stays out of 9–10 because it’s a standard, well-understood method at small scale, not research or heavy engineering. Anchored to the rubric’s 7–8 band and those named ledger projects (G10).

Time is separate from difficulty: the code runs in about a second (measured Part A wall-clock 1.31s). Part A is up to 90 minutes of reading and running; Part B is 3–4 hours of your own work.

3. What you’ll be able to do after

Implement biased matrix factorization in numpy — user/item latent factors plus offsets — and train it by gradient descent, watching the error fall as it converges.
Beat a sensible baseline on rating prediction and prove it with held-out RMSE, rather than reporting training error.
Turn a rating predictor into a ranker: score candidates per user, recommend a top-k, and evaluate with precision@k.
Explain why ranking quality (precision@k) and rating accuracy (RMSE) are different goals, and why beating a popularity baseline is the bar that matters.

The finished result

By the end of Part A you’ll have trained a recommender from scratch and scored it, honestly, on ratings the model never saw — against the per-user-average baseline it has to beat:

baseline RMSE (predict each user's average):        1.374
train RMSE over 500 gradient steps:                 1.317 -> 0.326
matrix-factorization RMSE (held-out, never seen):   0.561   (2.45x better than baseline)

The latent factors learned which user likes which kind of movie, so the model predicts held-out ratings at less than half the error of guessing each user’s average.

4. Prereqs & time box

You can write functions and use numpy arrays comfortably, and you’ve trained at least one model before (you know what train/test and overfitting mean). You do not need prior recommender-systems knowledge or any calculus beyond “nudge things to reduce error” — the update rule is given and explained. This is the deep end of the set; give it room.

Where to write and run code

Two ways to run this project — pick either:

In your browser (no install). If the project page shows a ▶ Run button, click it to run Part A right here — nothing to set up. Part A uses only pandas and numpy, which run in the browser.
On your computer. The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements, and running Part A step by step — is covered once in Start Here. Come back here once python -c "import numpy" runs without error. Part B (writing your own solution) needs this local setup.

Time box

Part A — Guided Build: 60–90 minutes. Hard cap.
Part B — Your Turn: 3–4 hours. Hard cap.

5. Setup & data

Both datasets are synthetic, generated by one seeded command (byte-for-byte reproducible, git-ignored, never committed):

python data/make_data.py

That writes, for Part A (movies): data/ratings_a_train.csv (observed ratings to learn from) and data/ratings_a_test.csv (held-out ratings to score RMSE); and for Part B (books): data/ratings_b_train.csv (observed ratings) and data/candidates_b.csv (per reader, 40 books to rank — their true ratings hidden). The hidden taste structure and the Part B relevance are documented in data/make_data.py and used by the grader — not handed to you.

6. Part A — Guided Build

Run the full script and read along:

python data/make_data.py            # once
python part_a/train_recommender.py

Prefer the browser? If this page has a ▶ Run button, click it instead of the command above — Part A runs in your browser with nothing to install. The STEP output and every checkpoint below are identical either way.

The full, commented script is in part_a/train_recommender.py.

Step 1 — Load the ratings. The matrix is mostly empty.

Checkpoint: 400 users × 150 movies, ~9.7% of cells filled, 5,800 train / 1,449 held-out ratings.

Step 2 — The baseline to beat: predict each user’s average rating.

Checkpoint: baseline RMSE ≈ 1.37. A per-user average can’t tell one movie from another, so it can’t tell a film they’ll love from one they’ll hate.

Step 3 — Learn the latent factors by gradient descent. Each user gets a factor vector P[u], each movie a vector Q[i], plus per-user and per-movie offsets. Predict mu + bu[u] + bi[i] + P[u]·Q[i], compute the error on observed ratings, and nudge every factor down the gradient (with a regularisation term so factors don’t blow up).

Checkpoint: train RMSE falls from ~1.32 to ~0.33 over 500 steps — the factors are fitting the observed ratings.

Step 4 — Score on the held-out ratings the model never saw.

Checkpoint: matrix-factorization RMSE ≈ 0.56, a 2.45× improvement over the baseline. The factors captured which user likes which kind of movie — exactly what an average throws away.

Why this and not that

Why latent factors instead of, say, “users who rated X also rated Y”? Neighbour methods need overlapping ratings between items to compare them; on a 90%-empty matrix most pairs never co-occur. Factors compress every user and item into the same small space, so the model can relate a user to an item they share no neighbours with. It degrades far more gracefully as the matrix gets sparser.
Why the regularisation term? With four factors per user and only a handful of ratings each, the model can memorise the training ratings and generalise terribly. The reg penalty keeps factors small, trading a little training error for much better held-out error — you can see it by setting REG = 0 and watching the test RMSE rise.
Why score on held-out ratings, not training error? Training RMSE always improves with more steps and more factors; it tells you nothing about a rating the model hasn’t seen. The held-out RMSE is the only number that reflects the actual job — predicting the blanks.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
`pd.read_csv`	Load the observed ratings	read_csv
`np.random.default_rng(...).normal`	Random initial factors	default_rng
`np.add.at`	Scatter-add each rating’s gradient onto the right user/item rows	np.add.at (ufunc.at)
`DataFrame.groupby(...).mean`	The per-user-average baseline	groupby
Concept	Matrix factorization for recommender systems	Matrix factorization (Wikipedia)

If something looks off

The held-out RMSE (~0.56) is higher than the final train RMSE (~0.33) — did the model get worse? No — that gap is expected and healthy. A model always fits the ratings it trained on better than ones it never saw; the held-out number is the honest one, and at ~0.56 it still beats the ~1.37 baseline by 2.45×. (The train RMSE also barely moves after the first ~100 steps — that’s the model converging, not stalling.)
A checkpoint number doesn’t match. The data is seeded, so the numbers are exact. Re-run python data/make_data.py (you may have skipped or edited it), then re-run Part A.
Setup problems (Python not found, pip, FileNotFoundError, re-activating the venv): see the troubleshooting list in Start Here.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Different catalogue, different job. A bookshop has ratings in data/ratings_b_train.csv and, for each reader, 40 candidate books to consider in data/candidates_b.csv. You won’t predict a star rating this time — you’ll rank: for each reader, put the books they’ll love on top and recommend the top 5.

Same core skill as Part A (learn latent factors from ratings), now aimed at ranking. A ready-made trainer is provided in part_b/_mf.py — the Part A model, packaged — so you can spend your effort on the ranking, not on re-deriving the maths.

Acceptance criteria (checkable)

Fill in part_b/starter.py, then:

python part_b/starter.py            # writes part_b/out/
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

Your solution writes into part_b/out/:

recommendations.csv — user, rank, item (your top-5 books per reader).
report.json — users_served, k, precision_at_k, achieved_metric.

You’re done when the tests pass. The grader knows each candidate’s true relevance (a held-out rating ≥ 4), recovered from the seed, and scores your top-5 with precision@5. It requires valid recommendations (from each user’s candidates, ≤ 5 each) and precision@5 ≥ 0.60. The reference reaches 0.90; ranking by popularity manages only 0.33, so you must use the model’s per-user scores.

Constraints — what must be true, not how to get there

Recommend only from a reader’s own candidate list, at most 5, no duplicates.
Rank by your model’s per-user predicted score. Ranking everyone by an item’s overall popularity is the trap — it ignores individual taste and lands near 0.33.
Your reported precision_at_k must match your recommendations (the grader recomputes it).

Hints

Hint 1 — the shape of the problem

Train the model on ratings_b_train.csv (see _mf.py’s train_mf). Then, for each candidate (user, item) pair, ask the model for a predicted score. Ranking is just sorting those scores per user.

Hint 2 — from scores to recommendations

Group the candidates by user, sort each user’s candidates by predicted score (descending), and take the top 5. Write them out with rank = 1…5.

Hint 3 — why popularity fails (still not the code)

A globally popular book isn’t the best pick for a reader whose taste runs the other way. Precision@5 rewards personalised ranking: the same model score that predicted ratings in Part A also orders each reader’s candidates by their own taste. If your precision is stuck near 0.33, check that you’re sorting by the per-(user,item) score, not by an item-level average.

8. Self-check

You don’t need the answer key. You’re done when:

Your Part A held-out RMSE is well below the baseline, and you can explain why regularisation helped (try REG = 0 and watch).
Your Part B precision@5 is far above the ~0.33 popularity level, and you can say why personalised scores beat popularity.
You can state the difference between what RMSE and precision@k reward, and which one matches the recommender’s real job.
Rerunning the pipeline reproduces your numbers (fixed seeds).

9. Stretch

Tune it. Sweep the number of factors K, the regularisation, and the steps, and plot held-out RMSE. Find where more factors start to overfit.
Cold start. Some readers have very few ratings. Measure precision@5 for the lightest-rated users vs. the heaviest, and think about what you’d fall back on when a user is nearly new.
(Genuinely hard) Implicit feedback. Real logs often have only “bought / clicked,” not star ratings. Adapt the model to implicit signals (treat unobserved as weak negatives, weight the positives) and re-evaluate the ranking. It’s a different loss, and it’s what most production recommenders actually use.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

Lead with the one-liner anyone gets: “I built a recommender from scratch — it predicts held-out ratings at half the error of a per-user average, and its top-5 book picks are right 90% of the time vs. 33% for ranking by popularity.”
Show the training-RMSE curve and the held-out RMSE next to the baseline, and the precision@5 bar (yours vs. popularity).
State the insight in one sentence: predicting ratings and ranking a shortlist are different jobs, and beating the popularity baseline is the only result that counts.

Keep make_data.py, _mf.py, and the tests in the repo so anyone can reproduce your numbers from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Datasets	Synthetic — movie ratings (Part A); book ratings (Part B)
Data type	Synthetic (seeded, reproducible)
Generated by	`data/make_data.py` — `numpy.random.default_rng` (low-rank factors + noise)
Source / license	Not applicable — synthetic, generated in-repo; no third-party rights
Access / verification date	2026-07-18
Redistribution	Not applicable — generator committed, CSV output never committed
robots.txt / ToS	Not applicable — nothing is scraped or fetched

The data is synthetic and teaches the mechanic (learning latent factors and ranking with them); the numbers describe a generated world, not a real catalogue. The generative process — hidden low-rank taste factors plus noise — is documented in data/make_data.py.

Next up

Finished this one? Continue the AI & Machine Learning track:

Uplift Targeting: spend on the persuadable, not the sure thing → · Browse all courses