AI Hardware

Cerebras and AMD GPUs Split Inference in Production, Reporting 5× More Tokens per Megawatt

November 18, 20261 views阅读中文原文

Cerebras says its cloud now runs a split inference pipeline in production. AMD Instinct GPUs process the prompt, while Cerebras CS-3 wafer-scale systems generate the response. In one before-and-after comparison, Cerebras reported five times as many generated tokens per megawatt with no loss in token generation speed.

Why split the work?

Prefill processes the prompt in parallel and benefits from GPU compute. Decode generates one token at a time and is more sensitive to how quickly hardware can feed the model. Giving each phase its own hardware pool also lets an operator size the pools independently. The KV cache must travel between them, so the network and its failure handling become part of the serving design.

What the test measured

The November 14 comparison used one shared GPT-OSS-120B endpoint in Cerebras Inference Cloud. The 5× figure describes power efficiency on that endpoint. It does not establish the same gain for another model or higher generation speed for an individual user. Cerebras named AWS, Fireworks AI, Gimlet Labs, and General Compute as early partners. Its goal of reaching hundreds of megawatts by the end of 2027 is a future deployment target.

What to measure in your own trial

Compare tokens per watt, time to first token, and generation speed separately. Test under your own prompt lengths and concurrency, and include the effect of network contention, cache transfer failures, and operating two hardware pools.

Sources

Comments

0 published
Log inorsign up to leave a comment.

Loading comments...