From miles away throughout the desert, the Nice Pyramid seems like an ideal, easy geometry — a modern triangle pointing to the celebs. Stand on the base, nevertheless, and the phantasm of smoothness vanishes. You see huge, jagged blocks of limestone. It’s not a slope; it’s a staircase.
Keep in mind this the following time you hear futurists speaking about exponential development.
Intel’s co-founder Gordon Moore (Moore's Legislation) is famously quoted for saying in 1965 that the transistor depend on a microchip would double yearly. One other Intel govt, David Home, later revised this assertion to “compute energy doubling each 18 months." For some time, Intel’s CPUs have been the poster youngster of this legislation. That’s, till the expansion in CPU efficiency flattened out like a block of limestone.
In the event you zoom out, although, the following limestone block was already there — the expansion in compute merely shifted from CPUs to the world of GPUs. Jensen Huang, Nvidia’s CEO, performed a protracted recreation and got here out a powerful winner, constructing his personal stepping stones initially with gaming, then laptop visioniand not too long ago, generative AI.
The phantasm of easy development
Know-how development is filled with sprints and plateaus, and gen AI will not be immune. The present wave is pushed by transformer structure. To cite Anthropic’s President and co-founder Dario Amodei: “The exponential continues till it doesn’t. And yearly we’ve been like, ‘Effectively, this could’t presumably be the case that issues will proceed on the exponential’ — after which yearly it has.”
However simply because the CPU plateaued and GPUs took the lead, we’re seeing indicators that LLM development is shifting paradigms once more. For instance, late in 2024, DeepSeek stunned the world by coaching a world-class mannequin on an impossibly small finances, partially through the use of the MoE method.
Do you keep in mind the place you latterly noticed this system talked about? Nvidia’s Rubin press launch: The know-how consists of “…the newest generations of Nvidia NVLink interconnect know-how… to speed up agentic AI, superior reasoning and massive-scale MoE mannequin inference at as much as 10x decrease price per token.”
Jensen is aware of that reaching that coveted exponential development in compute doesn’t come from pure brute power anymore. Generally it is advisable shift the structure totally to put the following stepping stone.
The latency disaster: The place Groq suits in
This lengthy introduction brings us to Groq.
The largest features in AI reasoning capabilities in 2025 have been pushed by “inference time compute” — or, in lay phrases, “letting the mannequin suppose for an extended time period.” However time is cash. Customers and companies don’t like ready.
Groq comes into play right here with its lightning-speed inference. In the event you carry collectively the architectural effectivity of fashions like DeepSeek and the sheer throughput of Groq, you get frontier intelligence at your fingertips. By executing inference sooner, you’ll be able to “out-reason” aggressive fashions, providing a “smarter” system to clients with out the penalty of lag.
From common chip to inference optimization
For the final decade, the GPU has been the common hammer for each AI nail. You utilize H100s to coach the mannequin; you employ H100s (or trimmed-down variations) to run the mannequin. However as fashions shift towards "System 2" considering — the place the AI causes, self-corrects and iterates earlier than answering — the computational workload adjustments.
Coaching requires huge parallel brute power. Inference, particularly for reasoning fashions, requires sooner sequential processing. It should generate tokens immediately to facilitate complicated chains of thought with out the consumer ready minutes for a solution. Groq’s LPU (Language Processing Unit) structure removes the reminiscence bandwidth bottleneck that plagues GPUs throughout small-batch inference, delivering lightning-fast inference.
The engine for the following wave of development
For the C-Suite, this potential convergence solves the "considering time" latency disaster. Take into account the expectations from AI brokers: We wish them to autonomously guide flights, code total apps and analysis authorized precedent. To do that reliably, a mannequin would possibly must generate 10,000 inner "thought tokens" to confirm its personal work earlier than it outputs a single phrase to the consumer.
-
On an ordinary GPU: 10,000 thought tokens would possibly take 20 to 40 seconds. The consumer will get bored and leaves.
-
On Groq: That very same chain of thought occurs in lower than 2 seconds.
If Nvidia integrates Groq’s know-how, they clear up the "ready for the robotic to suppose" drawback. They protect the magic of AI. Simply as they moved from rendering pixels (gaming) to rendering intelligence (gen AI), they might now transfer to rendering reasoning in real-time.
Moreover, this creates a formidable software program moat. Groq’s largest hurdle has at all times been the software program stack; Nvidia’s largest asset is CUDA. If Nvidia wraps its ecosystem round Groq’s {hardware}, they successfully dig a moat so extensive that rivals can’t cross it. They’d provide the common platform: The very best atmosphere to coach and essentially the most environment friendly atmosphere to run (Groq/LPU).
Take into account what occurs whenever you couple that uncooked inference energy with a next-generation open supply mannequin (just like the rumored DeepSeek 4): You get an providing that may rival at the moment’s frontier fashions in price, efficiency and velocity. That opens up alternatives for Nvidia, from straight coming into the inference enterprise with its personal cloud providing, to persevering with to energy a rising variety of exponentially rising clients.
The subsequent step on the pyramid
Returning to our opening metaphor: The "exponential" development of AI will not be a easy line of uncooked FLOPs; it’s a staircase of bottlenecks being smashed.
-
Block 1: We couldn't calculate quick sufficient. Resolution: The GPU.
-
Block 2: We couldn't prepare deep sufficient. Resolution: Transformer structure.
-
Block 3: We are able to't "suppose" quick sufficient. Resolution: Groq’s LPU.
Jensen Huang has by no means been afraid to cannibalize his personal product strains to personal the longer term. By validating Groq, Nvidia wouldn't simply be shopping for a sooner chip; they might be bringing next-generation intelligence to the plenty.
Andrew Filev, founder and CEO of Zencoder
