AI 101: The Inference Chip Wars – MatX, Taalas, and the Cracks in the GPU Era

From NVIDIA Vera Rubin to model-as-hardware, and why “inference chips” are no longer one category

Quick answer: What are the Inference Chip Wars?

The Inference Chip Wars are the shift from GPU-only AI infrastructure to inference-specialized hardware competing on cost per token, latency, power efficiency, and context handling. NVIDIA Rubin extends the GPU baseline with a rack-scale platform, while MatX bets on programmable LLM-first acceleration and Taalas bets on model-as-silicon for stable, high-volume workloads. The right decision is workload-specific: fast-changing multi-model stacks usually favor GPUs, while stable inference at scale can favor specialization.

Subscribe for weekly operator-grade AI systems analysis:
https://www.turingpost.com/subscribe

Most AI today runs on general-purpose hardware. To be more concrete: GPUs. For years, it felt borderline unrealistic to crack the ecosystem NVIDIA built around them. Then something interesting happened, and it happened fast.

While this year is still young, we’ve got some really exciting development in the hardware news. And it’s focused on inference. This January, at CES, NVIDIA introduced its next platform, Vera Rubin, as a rack-scale AI supercomputer made out of six chips, tuned for inference-heavy, communication-heavy workloads. Last week, Taalas raised $169 million and is demoing a product that takes the most extreme inference bet in the market: instead of running a model on hardware, it turns a specific model into hardware (model-as-hardware concept). And only yesterday, MatX raised a $500 million Series B for MatX One, an LLM-first accelerator that is ambitious on paper and still a dark horse in practice because it is not shipping yet (but has a giants backing it).

All together it demonstrates that the competition now happens on the operational realities of inference: latency, cost per token, power, and the physics of moving parameters and KV cache around. GPUs remain the baseline. The crack is in the assumption that one baseline will serve every inference regime forever.

In today’s episode, we will do three things:

1. Update the GPU story: Blackwell was the flagship for the previous wave. Vera Rubin is the new reference point, and it is a platform play.

2. Explain why inference is the opening challengers can attack, even when training remains GPU-heavy.

3. Use Taalas and MatX as two opposite bets on what comes next, then place them in the broader landscape of inference-specialized hardware.

If you want the takeaway upfront: MatX is a bold programmable challenger with a long runway. Taalas is the most different idea in the market, and it deserves a closer look because it forces you to ask a hard question: how stable does a model need to be before it makes sense to etch it into silicon?

Why inference became the battlefield

Training is the glamorous part of the AI lifecycle. It is also the part that looks the most like classic High-Performance Computing (HPC): huge matrix multiplies, enormous clusters, performance measured in throughput at scale.

Inference is where the invoice shows up: once a model ships, you run it millions of times in the messy, user-driven world that benchmarks barely approximate. Prompts get weird, context windows bloat, and the system stack keeps accreting retrieval, tool calls, and multi-agent coordination, each layer adding latency, cost, and fresh new ways to fail. Even a “single response” can turn into a chain of generation steps and intermediate reasoning, which changes what you are paying for.

This shift matters because the bottlenecks shift too. Token generation is sequential at the user level. You can parallelize many things, but you still produce outputs token by token. The model’s “knowledge” lives in its weights, and inference constantly streams weights and reads and writes KV cache. When the workload is constrained by data movement, a design that reduces data movement can beat a design that does more compute, even if the compute is more flexible.

NVIDIA’s own Rubin messaging makes this explicit: inference is now communication-heavy and system-bound, so the platform story centers on rack-scale interconnect, predictable latency, high utilization, and infrastructure for context and data movement.

(If you want to refresh you knowledge about chip types, here is our full guide on AI hardware, where we explained how the main types work)

The GPU era, updated: Vera Rubin is the new baseline

A GPU is still a simple idea at heart: lots of parallel compute cores, fed by high-bandwidth memory, executing a program compiled from software. Inside a modern data-center GPU, you will find billions of transistors etched onto one or more silicon dies, arranged into many lightweight processing cores. Those cores sit next to a hierarchy of on-chip SRAM – registers, shared memory, and large L2 cache – which acts as a high-speed staging area and tries to catch reuse before the GPU has to reach out to external memory.

The cores are also paired with stacked High-Bandwidth Memory (HBM), usually sitting on the same package. HBM itself is built from vertically stacked DRAM layers connected through a silicon interposer so data can move quickly between memory and compute.

When you run a language model on a GPU, the model’s weights live in that memory. The GPU keeps fetching them as it generates tokens. Intermediate results shuttle between SRAM and HBM. Attention reads and writes its KV cache. It is a powerful, flexible pipeline, and it is also why inference can become expensive: moving the model’s own “stuff” is often the limiting factor.

Why GPUs won (and keep winning)

First, they fit the math of deep learning. Matrix and tensor operations parallelize well, and GPUs are built to split a big job into thousands of smaller tasks and compute them simultaneously.

Second, they are programmable. A model is software in the GPU paradigm. You can run many models on the same hardware, swap checkpoints, tune kernels, change frameworks, and keep your infrastructure useful while the model landscape mutates.

Third, they come with an ecosystem. NVIDIA’s software stack (CUDA) has been a compounding advantage for over a decade. Even when a competing chip has an interesting architecture, it still has to earn its place in real deployment pipelines.

Vera Rubin changes the unit of thinking

Vera Rubin is NVIDIA treating the rack as the unit of performance and efficiency. NVIDIA describes the Rubin platform as “six new chips” that together form an AI supercomputer, and positions the flagship Vera Rubin NVL72 rack as a rack-scale accelerator within a larger AI factory. Production ramps in the second half of 2026, which is also why “Blackwell vs challengers” already reads like last season’s debate.

Here are the Rubin facts that matter for an inference-centric conversation:

• Rubin GPU: 50 PFLOPS inference performance per Rubin GPU and 288GB HBM4 per package, designed around low-precision inference formats.

• Rubin NVL72: the rack-scale system integrates 72 Rubin GPUs and 36 Vera CPUs, alongside ConnectX-9 SuperNICs, BlueField-4 DPUs, and NVLink 6 switching as the scale-up fabric.

• NVLink 6 scale-up: 3.6 TB/s NVLink bandwidth per GPU and up to 260 TB/s of NVLink bandwidth per rack.

• Cost-per-token framing: NVIDIA claims up to a 10× reduction in inference token cost compared with Blackwell in its platform announcement, and the developer blog shows a cost-per-token comparison for long-context, reasoning-heavy inference workloads.

• Context as infrastructure: the Rubin platform announcement includes an Inference Context Memory Storage Platform built around BlueField-4, designed to store and manage inference context for multi-turn and agentic workloads.

This doesn’t make Blackwell “bad.” It makes Vera Rubin a clearer description of where NVIDIA believes the next bottleneck is: system-level memory and interconnect, sustained utilization, and the practical conversion of megawatts into usable tokens. It also clarifies what challengers have to do: they do not need to beat Rubin everywhere. They need to dominate a specific inference regime and make the tradeoffs acceptable.

What about newcomers? Taalas and MatX

Taalas: when the chip becomes the model

Taalas is one of the rare cases where a hardware company forces you to re-evaluate what “deployment” even means. Their bet is simple to state and hard to execute →