What machine learning actually does

Start with the thing that makes all of it strange: nobody writes the rules.

That's the whole departure. In ordinary programming a person works out what should happen and states it — if this, then that — and the machine follows the instructions. Machine learning throws the instructions away and supplies examples instead, then lets a search process work out the rules for itself. What comes out the other end usually works. Very often nobody can say precisely why.

Everything else — the architectures, the jargon, the scale, the arguments — is refinement of that single move. Get the move straight and both the successes and the failures become far easier to predict, because they come from the same place.

The basic move

Take spam filtering, the standard teaching example, and do it the old way first. You'd sit down and write conditions. Certain words. Certain senders. Too many capitals. The program applies your list, and every time spammers change tactics you go back and edit it.

The other way round, you don't write any conditions. You collect a large pile of messages, each one labelled spam or not spam, and hand the pile to an algorithm whose job is to find whatever statistical differences separate the two heaps. What it hands back is a model: something that takes a message it's never seen and produces a score.

You didn't write the rules. In most cases you can't state them afterwards either.

That's not a defect of one technique, it's the trade you've made. You've swapped a system you can read for a system you can only test, and you did it because the rules you'd have written by hand would have been worse.

What training actually consists of

"Training" sounds like teaching. It isn't. It's search.

Inside the model is a very large collection of numbers — the parameters — that start out meaningless. The model uses them to make a prediction for one of your labelled examples, and something called a loss function measures how badly wrong the prediction was. That single number is the whole of the feedback. Then the numbers get nudged a little in whichever direction would have made the loss smaller, and the process repeats, millions upon millions of times.

That's it. There's no comprehension anywhere in the loop, and no goal beyond making one number go down. The reason it's usually described as learning is that the vocabulary got borrowed early on and stuck, and the borrowed words carry implications the mathematics doesn't support.

Two consequences fall straight out. The model can only improve on what the loss function measures, so anything you didn't put in the loss is invisible to the process, however much you care about it. And whatever makes the loss go down will get exploited, whether or not it's what you meant — a machine-learning restatement of Goodhart's law, which says that any measure adopted as a target stops being a good measure.

Training and using are different activities

These get run together constantly, and separating them clears up a lot.

Training is the expensive search: many passes over the data, enormous computation, and it happens once. Inference is what happens afterwards, when you feed something in and get an answer out. At that point the parameters are frozen. The model is a fixed function — large, complicated, entirely static.

So the ordinary intuition that a system is "learning from you" as you use it is, by default, wrong. Your interactions may well be logged, and they may be used to train a future version, but that's a separate project undertaken by people on a schedule. Nothing is quietly updating itself while you type.

Which explains something people find odd — that a system can make the same mistake in exactly the same way for months. Of course it can. It's the same frozen function every time.

The examples are the product

Because the pattern comes out of the examples, the examples determine what gets found. This isn't a caveat at the end of the manual. It's the main event, and most of what goes wrong traces back to it.

Train on data that under-represents a group and the model will perform worse for that group, because it had less to go on and nothing in its objective told it to care. Train on historical decisions and it'll reproduce the patterns in those decisions, including the ones nobody would defend if they were written down as a rule. The system has no notion of fairness that it could be violating. It found what was reliably present in what it was shown, which is the only thing it was ever built to do.

There's a further problem underneath, which is that labels aren't facts. Someone decided what counted as spam, or as a defect, or as a good outcome. Ask two people to label the same ambiguous cases and they'll disagree at a rate that would embarrass most published accuracy figures. The model treats those labels as ground truth because it has nothing else.

Which is why "the algorithm decided" is usually a dodge. The algorithm found a pattern in a dataset that people chose, gathered, cleaned and labelled, and optimised an objective that people wrote. The decisions were made earlier, by humans, in rooms.

It finds correlation, and correlation is all it finds

A model learns which things go together. It has no account of why, and it can't tell a real relationship from an accidental one that happened to hold across every example it saw.

The oldest version of this problem isn't computational at all. In Berlin around the turn of the twentieth century a horse called Hans appeared to do arithmetic, tapping out answers with a hoof, and he was right often enough to draw crowds and a formal investigation. In 1907 the psychologist Oskar Pfungst worked out what was happening: Hans was reading tiny involuntary changes in the posture and expression of whoever knew the answer, and stopping when they relaxed. Put the questioner behind a screen, or ask them something they didn't know, and the arithmetic evaporated. The horse had learned something real and useful. It just wasn't arithmetic.

Models do this constantly, and the examples aren't exotic. Researchers demonstrating the point built an image classifier that separated wolves from huskies almost perfectly and turned out to be detecting snow, because the wolf photographs mostly had snow in them. Medical imaging models have been shown to pick up on which hospital or which scanner produced an image, because different sites treat different populations and the equipment leaves traces — so the model quietly learns to identify the hospital and gets the right answer for a reason nobody wanted.

You'll also hear a story about a tank detector that had really learned to spot sunny weather. Treat that one as folklore; it's been retold for decades with no dependable source. The documented cases are damning enough without it.

These aren't rare failures. They're the default risk, because a shortcut that works is exactly what the training process is designed to find. And the usual safeguard doesn't catch them: if the same shortcut is present in the data you held back for testing, the test will confirm everything is fine.

Overfitting, and the split that's meant to prevent it

Building a model that performs perfectly on data it has already seen is trivial. It can just memorise. That's overfitting, and it's worthless, because the entire point is performance on cases nobody has seen yet.

The tension is structural: a model flexible enough to capture real structure is also flexible enough to memorise noise, and from the inside those two look identical. Much of the practical craft is managing that — holding back data the model never touches during training, limiting complexity deliberately, stopping before memorisation sets in.

Where the split gets done carelessly the resulting numbers are meaningless, and they're usually impressive, which is a bad combination. Near-duplicate items landing on both sides. A patient contributing images to the training set and the test set. A time series shuffled at random so the model gets to see the future while predicting the past. That last one is spectacularly common and produces results that look like a breakthrough right up until the system meets a real week.

So any accuracy figure should prompt a second question. Accurate on what, and had it seen that before?

The world moves, and the model doesn't

Here's the failure that gets the least attention and causes the most trouble in deployment.

A model is a description of the conditions that produced its training data. If those conditions hold, it works. If they drift, it degrades — and the degradation is silent, because the model has no way of noticing that the world has changed. It'll carry on emitting confident answers about a situation it was never shown.

Sometimes the shift is abrupt, as when behaviour changes suddenly across a whole population and every forecasting system trained on the previous normal starts producing nonsense at once. Usually it's slower: a sensor drifts, a supplier changes a process, a form gets redesigned, people gradually alter how they phrase things. Nothing dramatic happens on any given day. The model just quietly stops being about the present.

Which is why deployment isn't the end of the work. Systems that hold up are monitored against fresh outcomes, and retrained when the gap opens. Systems that get built, validated once and left alone will decay, and nobody will be able to say precisely when it started.

Why it's confidently wrong

Model outputs come with numbers attached, and those numbers get read as confidence. They generally aren't confidence in any sense you'd want.

A classifier is built to pick the best available answer from the options it was given. Show it something outside all of them and it doesn't return "none of these" — that isn't in its vocabulary — it returns whichever option it finds least unlike the input, and the score can be high. The score describes an internal ranking, not a probability that the answer is right, and the two only line up when the input resembles the training data.

The same thing is behind adversarial examples, which are worth knowing about because they're so unintuitive. Alter an image by a tiny amount, calculated deliberately, and a classifier's answer flips completely while a human sees no change at all. That's been reproducible for over a decade. Whatever the model has locked onto is evidently not what we're looking at.

For systems that generate text or images the mechanism is a cousin of this. Such a model is optimised to produce output that fits the patterns of its training data. Plausibility and truth are strongly correlated in that data without being the same property, and the optimisation can't distinguish them. So the output is fluent, well-formed and confident whether or not it's correct, and the errors are hardest to spot exactly where you're least equipped to check.

Fluency isn't evidence. It's the thing being optimised.

Accuracy is usually the wrong number

One number deserves singling out, because it's quoted more than any other and it's often meaningless.

Suppose you're detecting something that occurs in one case out of every thousand. A model that says "no" every single time, without looking, is 99.9 per cent accurate. It's also completely useless, and it would pass a careless review.

Rare events break accuracy as a measure. What matters instead is the shape of the mistakes: of the cases you flagged, how many were real, and of the real cases, how many did you catch. Those two pull against each other — flag more and you catch more real ones while flagging more rubbish — and where you set that balance is a judgement about consequences, not a technical decision. A missed fraud and a wrongly frozen account aren't the same kind of wrong, and no accuracy figure will tell you which you're buying.

What "learning" doesn't mean

The vocabulary of the field was borrowed from psychology and biology, and it does damage.

Neural networks are called neural because the earliest versions were loosely inspired by connected neurons, with "loosely" doing an enormous amount of work — the resemblance is a diagram, not a mechanism. Units in layers pass numbers forward, each connection carrying an adjustable weight, and training nudges the weights. That's the extent of it.

What networks genuinely add is worth stating plainly, because it's a real achievement. Older approaches needed a person to decide in advance which features of the data mattered, which is slow and limited by whatever that person happened to think of. Deep networks derive their own intermediate representations from the raw input, building up from simple regularities to complicated ones across the layers. That's why they took over vision, speech and language, and no amount of scepticism about the marketing changes it.

The bill arrives as opacity. Behaviour is smeared across an enormous number of weights, not one of which corresponds to a reason a person would recognise. There's a research field devoted to producing explanations, and the honest description of most of its output is that these are approximations built after the fact — a plausible account of what a much simpler model would have done, not a transcript of what happened. When a decision has to be justified or contested, that gap matters a great deal.

The field has over-promised before

A little history is a decent inoculation, because the current excitement is not the first.

Arthur Samuel coined the term "machine learning" in 1959, describing a program that played draughts and improved by playing against itself. Around the same time Frank Rosenblatt built the perceptron at the Cornell Aeronautical Laboratory, a machine that learned to recognise patterns by adjusting weights on connections from a grid of photocells — the same basic idea as everything since. The announcement was covered breathlessly, with newspaper reports relaying expectations that such machines would soon walk, talk, see, write and be conscious of their own existence.

They didn't. In 1969 Marvin Minsky and Seymour Papert published a careful analysis showing what a single-layer perceptron fundamentally could not compute, and funding drained away for years. The capability that eventually arrived needed layers, a practical method of training them — backpropagation, popularised in 1986 — and then several decades of hardware.

The pattern repeats because the demonstrations are genuinely impressive and the extrapolation from a demonstration to a capability is where everyone goes wrong. A system that does something striking under the conditions it was built for tells you very little about how it behaves when those conditions slip.

Where that leaves you

Two opposite mistakes are on offer and both are easy to make.

The first is dismissal — pattern-matching over big piles of data, nothing to see. That's hard to hold given the results. Systems built this way have produced real advances in protein structure prediction, translation, speech recognition and medical imaging, and the outputs are useful whatever is or isn't happening inside.

The second is treating fluent, accurate-looking output as evidence of understanding. That's the more expensive error, because it's the one that leads to putting a system somewhere its failure modes matter and then not watching it.

The position that survives contact with the actual machinery sits between the two. These are powerful statistical tools whose behaviour is determined by the data they were shown and the objective they were given. When one does something baffling, the answer is nearly always in the examples — and the useful question isn't what the model thinks, because it doesn't, but what it was rewarded for finding.