So what makes you think Strix Halo with such a weak GPU and slow memory bandwidt...

aurareturn 3 months ago | parent | context | favorite | on: AMD's Strix Halo under the hood

So what makes you think Strix Halo with such a weak GPU and slow memory bandwidth can handle 50k context with a usable experience for a 32B model?

Let's be realistic here.

The compute, bandwidth, capacity (if 128GB) are completely imbalanced for Strix Halo. M4 Pro with 64GB is much more balanced.

Tepix 3 months ago [–]

You're probably right. However with sparse models and MoE, 128GB may be useful.