Mixture of Experts model is likely the most significant.

And the scale of everything. GPT3 embedding vectors are around 12,000, vs 768 shown here.

I was curious and the 12k figure closely approximates the median synapse dimensionality of human neurons. Maybe we don’t need much more.