You get much higher bandwidth (at lower power cost) between chiplets than if you used a multi-chip design or even an interposer.
The drawback is that you suffer a really interesting thermal problem and have to figure out what to do with the wafer space that doesn't fit in your square -- probably creating lower-scale designs that you sell.
The second drawback is that you can't really match your available computation to memory in the same way you can with a more conventional GPU. So you have to be able to split your model across chips and train model-parallel. The advantage is that model-parallel lets you throw a heck of a lot more computation at the problem and can help you scale better than using only data parallelism.
Model-parallel training is typically harder than data-parallel because you need high bandwidth between the computation units. But that's exactly what Cerebras's design is intended to provide.
You also have a yield management issue, where you have to build in the capability to route around dead chips, but that's not too nasty a technical detail. But if your "chip"-level yield (note that their chip is still a replicated array of subunits) is too low, it kills your overall yield. So they're going to be conservative with their manufacturing to keep yield high.
It's not obviously broken, but it's certainly true we need benchmarks -- but not just benchmarks, time for people to come up with models that are optimized for training/inference on the cerebras platform, which will take even longer.
Why even make the final giant chip rectangular? I get that the exposure reticles are rectangular, but since this is tiling a bunch of connected chiplets, why not use the full area of the wafer?
The drawback is that you suffer a really interesting thermal problem and have to figure out what to do with the wafer space that doesn't fit in your square -- probably creating lower-scale designs that you sell.
The second drawback is that you can't really match your available computation to memory in the same way you can with a more conventional GPU. So you have to be able to split your model across chips and train model-parallel. The advantage is that model-parallel lets you throw a heck of a lot more computation at the problem and can help you scale better than using only data parallelism.
Model-parallel training is typically harder than data-parallel because you need high bandwidth between the computation units. But that's exactly what Cerebras's design is intended to provide.
You also have a yield management issue, where you have to build in the capability to route around dead chips, but that's not too nasty a technical detail. But if your "chip"-level yield (note that their chip is still a replicated array of subunits) is too low, it kills your overall yield. So they're going to be conservative with their manufacturing to keep yield high.
It's not obviously broken, but it's certainly true we need benchmarks -- but not just benchmarks, time for people to come up with models that are optimized for training/inference on the cerebras platform, which will take even longer.