The title is usual clickbaity Register but the content is good enough.
TL;DW:
* The improvements don't come from transistor density but other tricks like putting 2 chips together, using smaller 4bit format, more on-chip memory, 2x memory bandwidth
* And Nvidia is packing more GPUs in the same rack and consuming unheard of amounts of power, with a huge toll on datacenter infra
And it seems all these have growth limits. No more 2x every year or even every other year.
Everything has growth limits. For me Moore's law is number of transistor per dollar, not number of transistor per unit area. And even if we don't have any technical progress that number will decrease as the fab would recoup the R and D costs and the competition will increase after technical saturation.
We would likely get 2x transistor per dollar every two year for quite a long time.
> The complexity for minimum component costs has increased at a rate of roughly a factor of two per year.
It's incredible how deeply ingrained this misinterpretation has been, even among the very technical, without ever examining its original source or phrasing.
Headline marketing "AI TOPS" is also based on whatever smaller format a new generation supports. E.g. doubling headline marketing performance from Ada to Blackwell by going from FP8 to FP4.
Don't forget "structured sparsity" where two out of every four weights must be zero. Another trick that Nvidia still happily assume for the sake of a graph or an headline figure.
There's a lot of progress on optical computing recently, with some of its proponents suggesting they could get up to 100x speedup on some things like LLMs relative to current chips. If they ever manage to deliver then we'll see a lot of 2x or more improvements.
Yes, exponential growth gets quite large quite fast. Think about how a $300 graphics card today competes against one from 15 years ago. They're orders of magnitude faster, more memory, etc.
TL;DW:
And it seems all these have growth limits. No more 2x every year or even every other year.