The least speculative: PPUs will be converted from capped profit to unlimited profit equity shares at the benefit of PPU holders and at the expense of OpenAI the nonprofit. This is why they are doing it.
You missed the part where OpenAI the nonprofit gives away the value that’s between capped profit PPUs and unlimited profit equity shares, enriching current PPUs at the expense of the nonprofit. Surely, this is illegal.
Yes and no. It sounds like the capped profit PPU holders will get to have their units convert 1:1 with unlimited profit equity shares, which are obviously way more valuable. So the nonprofit loses insanely in this move and all current investors and employees make a huge amount.
But these rumors were said and talked about several days. But no big options trade was made before the actual day of announcement. That’s why it’s telling.
And the practical flops always end up lower. As an example a V100 has 125 according to spec, but the ideal case is more like 100 and non-ideal like 60.
Not yet, but since people seem to want this it’s at the top of our roadmap. If you train a model now and want to get the weights, just message us and we‘ll give them to you ;)
About as soon as GPT-4 came out I said that OpenAI was doomed on the trajectory it was on because they could not afford to develop a GPT-5, GPT-6, etc.
Real innovation comes out of doing a lot of experiments and that means doing experiments quickly with the resources you have. So you do most of your experiments with non-frontier models, enough to make a good prediction of what would happen if you maxxed out your model size, then you go big. That's how you make everyone else have a "DeepSeek moment".
A company like Apple wants to pick something on the frontier and keep advancing on a straight line. Works great if you want to make an M1, M2, M3, ... ARM chip but that's not how progress works in AI today.
You are deeply misunderstanding what the KV cache referred to here is. It’s not for storing data. This is the KV cache that’s part of the model to reduce quadratic compute complexity into linear for self attention. This is not stored on SSD - it’s in VRAM (or CPU if you’re not using a GPU)
They, in fact, mention inference kv cache as use case in readme. The most advanced kv caching uses hierarchy of gpu ram/regular ram/ssd. Seems like they were able to use their storage abstraction for last tier.
KVCache is a technique used to optimize the LLM inference process. It avoids redundant computations by caching the key and value vectors of previous tokens in the decoder layers. The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s
That's because DeepSeek uses MLA which apparently does allow offloading the KV cache. That doesn't apply to all models, particularly the open-weight models that are primarily GQA AFAIK.
Any models allow offloading KV cache, it’s not a matter of model architecture but only of the implementation. The only somewhat difference can be for non transformer models. But for all attention models it’s the same – blob of data per each token. It’s much worse for older models with MHA because their KV cache is just too big, and it’s better for DeepSeek because their KV cache is the smallest. But it’s alright for current generation of GQA models as well.
Are you sure about that? GQA applies self-attention to every KV cache entry. If you're offloading, then you're having to dynamically page in all the KV cache entries into the GPU which is quite slow since the CPU/GPU link only has so much bandwidth. My understanding is that MLA reduces the size of the KV cache & doesn't necessarily attend to every KV token at every step which is why offloading to disk works (i.e. most of the tokens can remain on disk without ever being loaded into the GPU).
Offloading in this case doesn’t mean keeping the kv cache on the disk/in storage all the time, it means keeping it there when request isn’t in process of generation. While request being generated kv cache is indeed in vram.
As for MLA - Deepseek is, just like others, attend to all historical tokens. The only difference instead of having actual KV entries it has lower dimension KV entries, which are being projected into full blown KV entries on the fly during attention. It’s similar to GQA, just instead of just duplication KV entries by size of groups it applies linear transformation.
Ah OK. So this is for resuming chat context cheaply. What I said is still correct - 3FS is not part of the inference flow & not relevant to the paper which is about optimizing the KV cache usage at runtime.
If they had experimented using a newer model (gemma 3, deepseek-1 7b, etc.) and reported better results, would that be because their newer baseline model was better than the llama 2 model used in the previous methods' experiments? A more comprehensive study would include results for as many baseline models as possible. But there are likely other researchers in the lab all waiting to use those expensive GPUs for their experiments as well.
Sure. But papers take a really long time to write and go through peer review. I think my paper on collaborative editing took about 4 months from the point where we were done writing to the point at which it appeared on arxiv.
This research was almost certainly done well before Gemma 3 and Deepseek were released.
> Is this some joke? They use Llama 2 7B? What year is it?
They use llama2 to demonstrate that their compression method works. There are potential cases:
1. The method works on all / most LLMs. In this case, it does not matter on which model they demonstrated the effect.
2. The method only works on llama2, but not on other models. Given that they published the code, I expect that people will quickly test the method on many other models, so we will know that soon. And yet - there would be a scientific significance even if it works only on llama2, as it would mean that there's some special and good in that architecture.
But I would bet it's #1 - the method works on most of the models and they just picked whatever they had already had code bindings to, to save the effort.
Deny deny deny. Those that have already drunk the kool-aid will believe your denial. Those that are too lazy to look or only get their info from one source will not know any different than your denial. The rest are just wrong from being in opposition anyways.
It works anywhere as long as you are large enough of an entity
reply