What machine learning can predict: a number, a category, a group, a preference, the next word
2026-11-18
Machine learning is not one technique. It is a family of ways to fill in a blank — and the shape of the answer decides the method.
| you want to predict | the shape of the answer | the method |
|---|---|---|
| how much? | a number | regression |
| which one? | a category | classification |
| what groups? | a group | clustering |
| what structure? | a simpler picture | PCA |
| what will you like? | a preference | recommendation |
| what comes next? | the next thing | generation |
Every one of them gets the same four slides: what the problem is, how the approach works, whether it helps, and one example followed all the way through.
An 8 × 8 image is 64 numbers — the features.
The digit 0–9 is the label. One row is one example.
We hide the answers for 30% of the data. The only honest score is on the 540 digits the model never saw.
A classifier is a function from 64 numbers to one of ten labels.
fit — learn from examples.
predict — answer for new ones.
score — check the answers.
A line, a tree, a clustering, a recommender: the same three calls.
Monday’s six verbs become these three.
The most common digit is 3 — 183 of the 1,797 examples.
A “classifier” that always answers 3 is right 10.2% of the time.
For a regression, the dumb baseline is the mean.
If 1% of your email is spam, “never spam” is 99% accurate and completely useless.
Accuracy alone is a liar. Every real model has to beat the dumb baseline first.
A restaurant wants to know how much this table will tip, before the bill is paid. The answer is a number, and the guess has to be close enough to plan with.
The dumb answer is the average tip — the dashed line. It is right on average and wrong for every individual table.
The answer we want is a number, and the one thing we know is another number. The simplest rule that turns one into the other is a line: two numbers describe it completely, and it can be checked against every table we have.
Anything fancier — a curve, a tree, a network — has to beat this line before it is worth the trouble. So the line is not a toy: it is the standard the rest have to clear.
Fit a straight line through the cloud:
tip ≈ 0.92 + 0.105 × bill
That formula is the model. Give it a bill, it gives back a tip:
| bill | the line says |
|---|---|
| $10 | $1.97 |
| $30 | $4.07 |
| $50 | $6.17 |
Two numbers describe the whole line:
Regression: predicting a number from other numbers, by fitting a rule that is as close as possible to the examples.
For every dot, the miss is the vertical gap to the line. Square each miss, so a miss above and a miss below do not cancel. Add them up.
The line with the smallest total wins. That is all “least squares” means.
Against the mean, the line earns its keep:
| typical miss | |
|---|---|
| guess the mean | $1.38 |
| use the bill | $1.02 |
R² = 0.46: the line explains 46% of the variation in tips. The other 54% is everything the bill does not know — the service, the weather, the party.
A model that cannot beat the mean has found nothing.
A table with a $31.85 bill. The line says the tip will be $4.27. The tip was $3.18 — a miss of $1.09, about the typical miss.
That is the whole method: one row in, one number out, and a miss you can measure.
Now the number is passengers next month, and one feature is the month. A trend alone misses the summer peak; the month fixes it.
Trend only: off by 59. Same month last year: 48. Trend + calendar: 37.
A classifier predicts a category: spam or not, sick or well, which digit.
The input is still a row of numbers. A digit is 64 pixels, each 0–16.
The model never hears the word “stroke”. It gets 64 numbers and has to find the pattern — and the same digit looks different every time somebody writes it.
The dumb answer here is “always say 3”, which is right 10.2% of the time.
We cannot write a rule that turns 64 pixel values into “3”. Nobody can say which pixels matter, and the same digit is written differently every time.
What we do have is 1,257 examples with the answers attached. So every method in this part works the same way:
Compare the new digit with stored examples, and let the comparison decide.
Three different comparisons, three methods: the nearest example, a series of pixel questions, and a crowd of question-askers.
Keep every training digit. To classify a new one, measure how far it is from each digit you already have, and copy the label of the closest.
Distance is the ordinary straight-line distance across the 64 pixels: add up the squared difference in each pixel, then take the square root.
The closest of the 1,257 training digits differs by a distance of 19 — under a third of a shade per pixel. Two handwriting samples that close should be the same digit.
One neighbour is a fragile answer: a single odd digit can decide everything. So take the k closest and let them vote — here, the ten closest.
Seven say 3. Three say 8. The answer is “3, with 30% doubt”.
Each box is a yes/no question about one pixel. Follow the answers down the branches; the leaf you land in is the answer, with the number of training digits that landed there.
It is built greedily: try every pixel and every threshold, keep the question that separates the two piles best, then repeat on each side with the digits that went that way.
Three questions deep, the tree lands in a pile of 614 training digits: 115 threes and 115 eights. It is a tie, broken by an arbitrary rule. It says 3 — and it is right by luck.
Let the tree keep splitting until every leaf holds a handful of digits, and the same 3 lands in a leaf of two training digits, both 8s. It says 8, with no doubt at all.
A tree three questions deep gets 47.6% — too simple. Unlimited depth gets 100% on the data it trained on and 84.3% on the data it never saw.
That gap is overfitting: the model learned the examples, not the pattern. The previous slide is the extreme case — a leaf with two training digits in it.
Grow a hundred trees. Each one sees a random sample of the rows — some digits twice, some not at all — and each split may only look at a random handful of pixels.
So each tree asks different questions. Three of them, on the same 3:
| tree | its first question | its answer |
|---|---|---|
| 1 | is pixel (4,4) dark? | 9 |
| 2 | is pixel (3,6) dark? | 3 |
| 3 | is pixel (2,5) dark? | 3 |
Two to one for 3. Let all hundred vote.
One deep tree gets 84.3%. A hundred deep trees, each on a different sample, average to 97.0% — a random forest.
The individual trees are still wrong as often as before. Their mistakes are just in different places, so the vote cancels them.
| model | what it is, in one line |
|---|---|
| nearest neighbour | copy the closest example |
| decision tree | twenty yes/no questions about pixels |
| random forest | a hundred trees voting |
The dumb answer is 10.2%; every real model is above 84%; the gap between the simplest and the fanciest is about 13 points.
The test digit is a 3. Ten neighbours vote, and this is the tally.
The answer is “3, with 30% doubt”.
The doubt is not decoration. It is a distribution over the ten digits — and the last part of this lecture is what you can do with one.
Four thousand 311 calls, each with a (latitude, longitude). Nothing else.
Nobody tells us which calls are Bronx and which are Brooklyn. There is no label to predict, and no answer to check against.
The question is: does this cloud have parts?
There are no labels here, so nothing can be right or wrong, and nothing can be scored. The only thing we can measure is how far apart two points are.
So “group” has to mean “points that are close to each other”, and the method has to be a way of finding clumps of closeness. Distance is not a choice we made; it is the only evidence in the file.
Two points are close if the straight-line distance between them is small.
A good set of groups is tight inside and far apart between: points in the same group should be near each other, and the groups should not overlap.
We choose the number of groups k. There is no correct k — only a defensible one.
Clustering: finding groups in data when nobody has told you what the groups are.
Drop k centres at random. Assign every point to its nearest centre. Move each centre to the mean of its points. Repeat until nothing moves.
Five groups, from nothing but coordinates.
The five patches are the city’s density, and they land close to the boroughs — nobody told it about geography.
They are not the boroughs: a cluster is a patch of dense calls, and a borough is a government. The model found structure, not meaning.
The curve shows how much spread is left after k groups. It always falls; the question is where it stops falling fast.
The bend is around 4–6. We chose 5 because the city has five boroughs — a judgement, not a law.
One address filed 14,362 complaints in November. The next filed 728. That is a 20× gap.
Same idea, one number per point: model what normal looks like, then flag what is far from it.
A digit is a point in 64-dimensional space. You cannot draw that, and neither can I.
Most of the 64 numbers are redundant. The spread of each pixel across all 1,797 digits is on the right: the border barely moves, so it cannot help tell one digit from another.
We want fewer numbers that keep most of the structure.
We have to keep fewer numbers, so we have to decide which ones to throw away. A column that is the same for every digit — the border — tells us nothing. A column that moves in step with another one is redundant.
So the numbers worth keeping are the directions in which the data actually varies. That is the whole idea of a principal component.
Centre the data, then find the direction in which it spreads the most. That is the first component — the longest shadow of the cloud.
The second component is the longest direction at right angles to the first.
Each component comes with a share of the variance: how much of the spread it carries.
Principal component analysis: replacing many correlated columns with a few new ones, built as weighted combinations of the originals.
Drop every digit straight down onto the first component: each one becomes a single number.
That is all a component does — it throws information away on purpose, and keeps the part that varies most.
Two numbers per digit instead of 64.
The digits form islands: similar digits land near each other. But the islands overlap — a 2-D shadow cannot separate ten handwriting styles.
Two components carry 28.5% of the spread, so this is a lossy summary, not the data.
Keep k components, then rebuild the image from them.
One component: 15% of the spread. Ten: 74%, and the digit is readable. Forty: 99%.
A recommendation predicts a preference: a rating, a click, a song.
The data is a matrix, not a table: users down one side, films down the other, and most cells are empty. 149 of 240 here.
The question is the missing cell: what would this user give this film?
We want to fill a cell nobody filled. There is no rule that turns a user and a film into a rating.
The only evidence in the file is what other people said about the same film — and the useful ones are the people whose taste already matches. So the method has to be: find those people, and average what they said.
Find the users whose ratings look most like yours, and let them vote on the film you have not seen.
Similarity is computed only over films both users rated — otherwise a user who rated everything would look similar to everyone.
Each user’s ratings are centred first: a harsh grader and a generous one can still agree.
For user 215 and Fight Club:
| neighbour | similarity | their rating |
|---|---|---|
| 581 | 0.94 | 4.5 |
| 211 | 0.78 | 5.0 |
| 471 | 0.76 | 4.0 |
| 419 | 0.70 | 5.0 |
| 111 | 0.62 | 5.0 |
Each neighbour’s rating is multiplied by their similarity; the total is then divided by the total similarity.
| neighbour | similarity × rating |
|---|---|
| 581 | 0.94 × 4.5 = 4.23 |
| 211 | 0.78 × 5.0 = 3.90 |
| 471 | 0.76 × 4.0 = 3.04 |
| 419 | 0.70 × 5.0 = 3.50 |
| 111 | 0.62 × 5.0 = 3.10 |
| total | 17.77 ÷ 3.80 = 4.68 |
A neighbour who agrees with you more gets more of the vote — that is the “weighted” part.
The nearest neighbour alone would have said 4.5. The crowd says 4.68.
That is the whole of user-user collaborative filtering.
Fight Club was predicted at 4.68; the actual rating was 4.5. We hid a rating we knew and got it back to within a fifth of a point.
The violet bars are films already rated — the check. The orange ones are the recommendations: films user 215 has not seen.
A new user has rated nothing. A new film has been rated by nobody.
Every method on the previous slides needs other people’s opinions, and for a brand-new user or film there are none. That is the cold start, and no amount of matrix arithmetic fixes it.
The fixes are not arithmetic: ask the new user a few questions, or use content features — genre, cast, text.
Some answers are sequences: the next character, the next word, the next frame of a film.
A single label is not enough here. After “lord,” there are many characters that could come next, and several of them are reasonable.
What we want is not one answer but the whole set of answers with their chances — a distribution.
Distribution: the possible answers, with the share of the time each one happens. A histogram is one; a list of percentages is one.
We want the next character. There is no rule that says what follows “lord,”.
The only evidence is what followed it last time. So we count, over every context in a million characters, and let the counts be the model. When we meet a context we have never seen, we fall back on how often each character appears at all.
Train on 1 MB of Shakespeare. For every six-character context, count which character came next.
This is one row of that table: after “lord,” the next character was “t” 18% of the time, “I” 17%, “w” 11%, and so on.
Fit means count. Predict means look up the row.
Predict takes the tallest bar: the answer is “t”.
Sample rolls a weighted die: take a bar at random, with taller bars more likely to win. This time it came back with “I”.
Same row, same numbers. One is a decision, the other is a draw.
Six characters of context, 65 possible next characters.
It gets the next character right 46.6% of the time. Guessing at random would be about 1.5%.
That is not a good language model by modern standards. It is a perfectly good classifier — and everything in the next two slides is the same table, read differently.
ROMEO: A gentle ears. Provost: I am disgracious lord, I will not out a word. PRINCE EDWARD IV: What she shall see us richly in the come to the people in a command. I was bad into a poor dukedom, through the southern lords, thou art a villain in love,
Predict a distribution → sample one character → append → repeat.
It is not retrieving Shakespeare. It is guessing, then guessing again, 245 times.
| this model | a modern language model | |
|---|---|---|
| context | 6 characters | ~100,000 tokens |
| choices | 65 characters | ~100,000 tokens |
| parameters | a few million counts | hundreds of billions |
| training text | 1 MB | trillions of tokens |
The operation is identical: P(next thing | context). Only the scale changed.
To tell things apart well, a model ends up learning how they are made.
Ask a narrower question: is this digit a 3?
There are 55 threes in 540 test digits. “Never a 3” is 89.8% accurate — and finds none of them.
Precision: of the digits we call 3, how many are 3? 82%
Recall: of the real 3s, how many did we find? 91%
The model sees numbers. Your table is not numbers yet.
| raw | encoded | why |
|---|---|---|
day = "Sat" |
day_Sat = 1, others 0 |
a model cannot multiply a word |
bill = 42.7 |
(bill − mean) / sd |
keep one big column from dominating |
smoker = "Yes" |
smoker = 1 |
two categories become one column |
"no heat in apartment" |
counts of 4,000 words | text becomes a row of numbers |
missing age |
median, or a “missing” flag | a blank is information too |
Most of the work in a real project is this column, not the model.
A model learns the world in the data, including its unfairness.
The model is not neutral because it is arithmetic.
| you want | the method | reach for it when |
|---|---|---|
| a number | regression | the answer is a quantity |
| a category | classification | the answer is one of a few labels |
| a group | clustering | there are no labels |
| a simpler picture | PCA | the table is wide and correlated |
| a preference | recommendation | the data is a users × items matrix |
| the next thing | generation | the answer is a sequence |
Start with the shape of the answer. The method follows.
The same three calls — fit, predict, score — run a line, a tree, a recommender, and a trillion-parameter model.
The lab: build the six, break the tree, sample the text.
CS 2010 · The Shape of a Guess