Why transformers outperform CNNs
This blog explains what made them completely replace CNNs for most applications.
In 2017, a team at Google proposed the transformer, a model built to translate sentences.[1]
Three years later, they beat CNNs for the first time at classifying photographs.[2]
Today, most AI systems have switched to transformers.
- Uber uses them to predict cab arrival times.
- Netflix uses transformers for movie recommendations.
- Meta uses transformers for news feed generation.
So why are transformers so popular? This blog explains what made them completely replace CNNs for most applications.
1. How a CNN works
How does a model classify a photograph of the Taj Mahal? Until 2020, the answer was almost always a convolutional neural network, or CNN.
Until recently, most image classifiers used to use convolutional neural networks. These networks assume sparse local connectivity, meaning every point of the input image is only connected with its nearby points.
A CNN slides a small window across the image. At each stop, it combines the pixels under the window into one number, such as their average brightness. Those numbers form a smaller image.
The window is called a filter, and the smaller image a feature map. Each layer slides its filters over the feature maps of the layer before it.
The Vision Transformer paper calls this an inductive bias, "baked into each layer throughout the whole model".[2] It helps a CNN learn from fewer images.
It also limits what one layer can relate. A layer reading the top of the dome doesn't read the pool below it.
2. How a vision transformer works
What if one layer could read the dome and the pool together? A transformer reads a sequence of tokens, and self-attention lets every token read every other one.
Making each pixel a token costs too much. A 224 by 224 image has 50,176 pixels, and attention grows with the square of that number.
So the Vision Transformer, or ViT, cuts the image into patches of 16 by 16 pixels. That gives 196 tokens in place of 50,176.
Each patch becomes a vector, a token, and a learned position vector is added to it. The tokens go into the bottom of a standard transformer encoder.
The main difference here is that global features are easier to identify.
The paper puts it in six words: "the self-attention layers are global". Some heads attend to most of the image in the very first layers.
3. Google: more data, less compute
Does that make ViT the better classifier? Only with enough images. Trained on ImageNet's 1.3 million images alone, ViT lands "a few percentage points below ResNets of comparable size".
Those few points are the price of the missing locality. A CNN gets it for free, and ViT has to learn it from the images. So Google pre-trained ViT on JFT, a set of 303 million images, and compared it with BiT-L, a ResNet trained on the same set.
| ViT-L/16 | BiT-L | |
|---|---|---|
| Architecture | Transformer | ResNet, a CNN |
| ImageNet accuracy | 87.76% | 87.54% |
| Pre-training cost in TPUv3-core-days | 680 | 9,900 |
The transformer was more accurate, and it took about a fourteenth of the compute. Averaged over five datasets, ViT used "approximately 2 to 4 times less compute to attain the same performance".
Finally, a classifier classifies this image, and this output tends to scale better as compared to convolution neural networks.
ResNets do better on small sets but plateau sooner, and ViT didn't saturate within the range the authors tried. I explained this paper in a one-minute reel.
4. Uber: features as tokens
Does the same hold away from images? Uber's arrival times are a table of numbers, not a picture.
For years, Uber refined its arrival times with XGBoost, one of the largest and deepest XGBoost ensembles in the world.[3] Then growing the data and the model with XGBoost "became untenable".
So the DeepETA team tested seven neural network architectures. An encoder with self-attention, a transformer, gave the best accuracy.
Here each token is one feature of the trip, such as the origin or the time of day. Self-attention lets every feature weigh every other one. The traffic reading is read in light of the time and the route.
There is no position embedding at all. The order of the features doesn't matter, unlike the patches of a photograph.
But the full transformer was too slow. DeepETA is Uber's busiest model, and it has to answer within a few milliseconds.
So Uber chose a linear transformer, which never builds the full attention matrix. With 40 features of 8 dimensions, the cost drops from 12,800 to 2,560 operations.
5. Netflix: one model for every row
Can one transformer replace many models? Netflix's home page used to be served by specialised models, one for "Continue Watching", another for "Today's Top Picks for You".[4]
Each was trained on its own, so an idea in one model was hard to carry to another. Most also read only a short window of a member's history.
Netflix is now moving its recommendation system to a transformer based foundation model, with the idea of condensing multiple specialized models into a single comprehensive model.
Instead, user interactions and activities are noted as tokens.
The model learns the way a language model does. It predicts the next interaction. That lets it train on hundreds of billions of interactions with no human labels.
And it scales the way a language model does. Netflix reports "consistent improvements observed as we increase data and model size". I covered this move in a reel too.
Latency limits it here as well. A request has milliseconds, and the context is held to a few hundred events. Sparse attention reaches that far.
6. Why the transformer wins
What do the three cases share? The model relates every input to every other one. It keeps getting better with more data.
A CNN's locality and XGBoost's trees help with less data and then plateau. A transformer learns those relations itself. It needs more data, and it keeps improving after the others stop.
Each team also paid for attention's cost in its own way:
| Uber | Netflix | ||
|---|---|---|---|
| What a token is | a patch of 16 by 16 pixels | one feature of the trip | one member interaction |
| What it replaced | ResNet, a CNN | XGBoost | many specialised models |
| How the cost was kept down | 196 patches in place of 50,176 pixels | a linear transformer | a context of a few hundred events, sparse attention |
So the transformer wins when there is enough data for it to learn what older models assumed. With a small dataset, the CNN's built-in locality still wins.
Summary
- A CNN connects each point of an image only to its nearby points, in every layer.
- A transformer's self-attention relates every token to every other token, from the first layer.
- Attention cost grows with the square of the number of tokens. ViT cuts an image into 196 patches of 16 by 16 pixels.
- Trained on 1.3 million images, ViT is a few points below a comparable ResNet. Pre-trained on 303 million, it beats BiT-L with 680 TPUv3-core-days against 9,900.
- Uber's DeepETA embeds each trip feature as a token, with no position embedding, and uses a linear transformer to answer within milliseconds.
- Netflix's foundation model treats member interactions as tokens and improves as data and model size grow.
Sources
- Attention Is All You Need, Ashish Vaswani and others, Google, 2017.
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Alexey Dosovitskiy and others, Google Research, October 2020; published at ICLR 2021.
- DeepETA: How Uber Predicts Arrival Times Using Deep Learning, Tanmay Binaykiya and others, Uber Engineering, 10 February 2022.
- Foundation Model for Personalized Recommendation, Ko-Jen Hsiao and others, Netflix Technology Blog, 21 March 2025.
- Transformers are outperforming CNNs in image classification, Gaurav Sen, YouTube, 22 January 2025.