In defense of Byzantines. Their rules and amazing diplomatic prowess is what let them be an empire for so long. The negative connotations to Byzantine comes from the negative perception the west had of them. Byzantines were very practical in regards to who they allied with.
The claims about extreme complexity of the Late Roman/Byzantine state came into the popular imagination by Enlightenment and other Western thinkers who were deeply biased against the late Romans due to a long history of cultural conflict. The OP is completely correct here, the reddit comment is an extremely incomplete story. It notes that the Roman state was more complex than other Medieval states (correct), but to say that it was "too" complex is a culturally based judgment, not a fact. The origin of the negative cultural judgment about that complexity waen't coming from the Romans, they were coming from the Franks, Venetians, and later Western Europeans who in large part were repeating the old prejudices going back to the schism, but also justifying their own conquest and abuses of of the Roman people.
I'm sure the children who watched their parents get murdered before they themselves were taken into slavery during the fall of Constantinople appreciated those rules and the alliances they supported.
Its interestine, when you say abstractions. Could you explain what you mean by abstractions in this context and what do you mean by the underlying fundamentals.
One example would be resonant circuits. Ok great you can build resonant circuits, but what for? The fundamentals to understand frequency responses came later in signals and systems. The application came much later when I learned about electric motors, which basically behave like low pass filters (resonant circuits) which enables us to use PWM to generate sine shaped current curves by switching the input voltage on and off. The voltage signal is smoothed by the LPF circuit that is the motors windings.
I think it would have helped me if we talked about the motor or other examples first, and then did some math to show how the resonant behavior can be useful.
Check out Oxide and Friends[0]! We've been doing it for several years now, and it's a much more flexible format that allows the team to be heard in its own voice -- and allows us to weigh in on whatever's on our collective mind.
To summarise, photosynthesis increases the amount of molecular oxygen in the atmosphere. It doesn’t create oxygen atoms. Those come, in the recent geological era, from inside the earth in the form of volcanic oxides. The paper suggests that something about the magnetic field influences the rate at which those oxides are belched into the atmosphere.
Are there any good sources that I can read up on estimiating what would be hardware specs required for 7B, 13B, 32B .. etc size If I need to run them locally? I am grad student on budget but I want to host one locally and trying to build a PC that could run one of these models.
"B" just means "billion". A 7B model has 7 billion parameters. Most models are trained in fp16, so each parameter takes two bytes at full precision. Therefore, 7B = 14GB of memory. You can easily quantize models to 8 bits per parameter with very little quality loss, so then 7B = 7GB of memory. With more quality loss (making the model dumber), you can quantize to 4 bits per parameter, so 7B = 3.5GB of memory. There are ways to quantize at other levels too, anywhere from under 2 bits per parameter up to 6 bits per parameter are common.
There is additional memory used for context / KV cache. So, if you use a large context window for a model, you will need to factor in several additional gigabytes for that, but it is much harder to provide a rule of thumb for that overhead. Most of the time, the overhead is significantly less than the size of the model, so not 2x or anything. (The size of the context window is related to the amount of text/images that you can have in a conversation before the LLM begins forgetting the earlier parts of the conversation.)
The most important thing for local LLM performance is typically memory bandwidth. This is why GPUs are so much faster for LLM inference than CPUs, since GPU VRAM is many times the speed of CPU RAM. Apple Silicon offers rather decent memory bandwidth, which makes the performance fit somewhere between a typical Intel/AMD CPU and a typical GPU. Apple Silicon is definitely not as fast as a discrete GPU with the same amount of VRAM.
That's about all you need to know to get started. There are obviously nuances and exceptions that apply in certain situations.
A 32B model at 5 bits per parameter will comfortably fit onto a 24GB GPU and provide decent speed, as long as the context window isn't set to a huge value.
Assuming the same model sizes in gigabytes, which one to choose: a higher-B lower-bit or a lower-B higher-bit? Is there a silver bullet? Like “yeah always take 4-bit 13B over 8-bit 7B”.
Or are same-sized models basically equal in this regard?
I would say 9 times out of 10, you will get better results from a Q4 model that’s a size class larger than a smaller model at Q8. But it’s best not to go below Q4.
But it’s still surprising they haven’t. People would be motivated as hell if they launched GPUs with twice the amount of VRAM. It’s not as simple as just soldering some more in but still.
They sort of have. I'm using a 7900xtx, which has 24gb of vram. The next competitor would be a 4090, which would cost more than double today; granted, that would be much faster.
Technically there is also the 3090, which is more comparable price wise. I don't know about performance, though.
VRAM is supply limited enough that going bigger isn't as easy as it sounds. AMD can probably sell as much as they get their hands on, so they may as well still more GPUs, too.
Go to r/LocalLLAMA they have the most info. There’s also lots of good YouTube channels who have done benchmarks on Mac minis for this (another good value one with student discount).
Since you’re a student most of the providers/clouds offer student credits and you can also get loads of credits from hackathons.
Generally, unquantized - double the number and that's the amount of VRAM in GB you need + some extra, because most models use fp16 weights so it's 2 bytes per parameter -> 32B parameters = 64GB
typical quantization to 4bit will cut 32B model into 16GB of weights plus some of the runtime data, which makes it possibly usable (if slow) on 16GB GPU. You can sometimes viably use smaller quantizations, which will reduce memory use even more.
You always want a bit of headroom for context. It's a problem I keep bumping into with 32B models on a 24GB card: the decent quants fit, but the context you have available on the card isn't quite as much as I'd like.
I've only recently started looking into running these models locally on my system. I have limited knowledge regarding LLMs and even more limited when it comes to building my own PC.
Are there any good sources that I can read up on estimiating what would be hardware specs required for 7B, 13B, 32B .. etc size If I need to run them locally?
Why are you posting this to very second comment? What do you mean by "actual science"? Like it's not based on chemistry? Boy, have I got a surprise pharmaceutical industry for you.
It seems you don't have a good understanding of either "therapy" and "science"
Calling other's ideas "hot air" is just the cherry on top.
reply