It's aimed at OpenAI's moat. Making sure they don't accumulate too much of one. No one actually has to use this, it just needs to be clear that LLM as a service won't be super high margin because competition can simply start building on Meta's open source releases.
The moat is all but guaranteed to be the scale of the GPUs required to operate these for a lot of users as they get ever larger, specifically the extreme cost that is going along with that.
Anybody have $10 billion sitting around to deploy that gigantic open source set-up for millions of users? There's your moat and only a relatively few companies will be able to do it.
One of Google's moats is, has been, and will always be the scale required to just get into the search game and the tens of billions of dollars you need to compete in search effectively (and that's before you get to competing with their brand). Microsoft has spent over a hundred billion dollars trying to compete with Google, and there's little evidence anybody else has done better anywhere (Western Europe hasn't done anything in search, there's Baidu out of China, and Yandex out of Russia).
VRAM isn't moving nearly as fast as the models are progressing in size. And it's never going to. The cost will get ever greater to operate these at scale.
Unless someone sees a huge paradigm change for cheaper, consumer accessible GPUs in the near future (Intel? AMD? China?). As it is, Nvidia owns the market and they're part of the moat cost problem.
> VRAM isn't moving nearly as fast as the models are progressing in size.
Models of any given quality are declining in size (both number of parameters, and also VRAM required for inference per parameter because quantization methods are improving.)
and this is why the LLM arms race for ultra high parameter count models will stagnate. It's all well and good that we're developing interesting new models. But once you factor cost into the equation it does severely limit what applications justify the cost.
Raw FLOPs may increase each generation but VRAM becomes a limiting factor. And fast VRAM is expensive.
I do expect to see incremental innovation in reducing the size of foundational models.
> VRAM isn't moving nearly as fast as the models are progressing in size. And it's never going to. The cost will get ever greater to operate these at scale.
It is in at least 2025. AMD (and Intel, maybe) will have M-Pro-Esque APUs that can run a 70B model at very reasonable speeds.
I am pretty sure Intel is going to rock the VRAM boat on desktops as well. They literally have no market to lose, unlike AMD which infuriatingly still artificially segments their high VRAM cards.
