Talk about AI hardware and one question always comes up: computers have had CPUs for ages, so why does AI need a GPU?
This piece explains the difference in plain terms, and why large-scale AI training runs almost entirely on GPUs. It’s the companion to the GPU gate and a starting point for The AI Hardware Supply Chain, End to End.
The Difference in One Line: Quality vs. Quantity
The key difference between a CPU (central processing unit) and a GPU (graphics processing unit) is the number of cores and how the work is split.
A CPU has a few to a few dozen very powerful cores, good at working through complex, varied tasks in order. A GPU has tens of thousands of relatively simple cores, good at running lots of the same kind of math at the same time.
Think of it this way: a CPU is a few PhD students. Each can solve very hard problems, but there are only a few. A GPU is a few thousand grade-schoolers. Each can only do simple arithmetic, but there are a lot of them. For a job like “solve the same kind of problem millions of times,” the grade-schoolers working together are amazingly fast.
Why AI Training Uses GPUs
It comes down to the shape of the work.
Under the hood, training an AI model is a huge amount of repetitive matrix math that can run at the same time (lots of numbers being multiplied and added). This “same move, millions of times” work is exactly what a GPU’s many cores do well together. A GPU computes a big chunk at once; a CPU’s few cores have to queue up in batches, which is much slower.
The CPU isn’t bad. This just isn’t its kind of work. That’s why training and inference for large models run almost entirely on GPUs or more specialized chips.
The CPU Isn’t Replaced: It’s a Division of Labor
In a real AI system, the CPU and GPU work as a team.
The CPU handles scheduling, program flow, logic decisions, and data preparation. The GPU gets the mass of parallel math done. The CPU is like a project manager, deciding who does what and when. The GPU is like a big crew of line workers, finishing the same kind of task at high speed. Take away either side and the whole system runs poorly.
More Specialized Than GPUs: TPUs and ASICs
A GPU is great at parallel math, but it’s still fairly general: it can run many kinds of computation. Companies that want lower cost and more speed build more specialized chips.
The TPU (Google’s Tensor Processing Unit) and the AI ASIC are chips built for specific AI work. On that job they are more efficient than a GPU, at the cost of being less flexible. They work alongside GPUs; neither replaces the other. To see how these chips compare, read What Are AI Chips.
Penchan’s summary
CPUs and GPUs are each good at different work. AI training is exactly the “lots of the same math” work GPUs do best, so large-scale training runs mainly on GPUs or specialized accelerators, while the CPU stays in the system to handle scheduling.