> It's the huge matrices where performance matters

I wouldn't say that, just look at CGI where a lot of effort is spent making fast 4x4 matrix multiplies. But yeah, a general matrix mult needs to scale, but a lot of the time you know what size you will run and you can optimize for that.