"GPT-3 is a marvel of engineering due to its breathtaking scale. It contains 175 billion parameters (the weights in the connections between the “neurons” or units of the network) distributed over 96 layers. It produces embeddings in a vector space with 12,288 dimensions."
I don't know much about AI, though I do know about programming, and to me this vaguely smells like "our program is so great because it has 1 million lines of code!"
Does the number of parameters, dimensions, etc, really have anything to do with how breathtaking and marvelous something like this is?
The whole schtick of GPT-3 is the insight that we do not need to come up with a better algorithm than GPT-2.
If we dramatically increase the number of parameters without changing the architecture/algorithm its capabilities will actually dramatically increase instead of reaching a plateau like it was expected by some.
"To the surprise of most (including myself), this vast increase in size did not run into diminishing or negative returns, as many expected, but the benefits of scale continued to happen as forecasted by OpenAI."
I love articles about the cutting edge of AI - because it always ends up putting the spectacular glory of nature front and center.
Any time you see an AI promo with really big numbers claiming to be close to replicating human cognitive functions, just remember that the human brain has 100 trillion synapses. So much of our discourse around AI right now is nothing more than chimpanzees scribbling on a wall with a crayon and calling it a "self portrait".
It _can_ be an analog to your million lines of code. But often times a given architecture won't scale to greater capabilities just by adding more parameters in the same pattern as before. (Oversimplifying, but) the signal from the training data gets weaker and weaker the "further away" the parameters are from it in the model. It can take actual ingenuity to figure out how to get those further layers in a network to contribute anything useful.
If nothing else, actually training an AI algorithm that large is an extremely large engineering challenge.
Googling around, it looks like most neural networks have somewhere in the neighborhood of tens of thousands of parameters. If nothing else GPT-3 is much, much bigger than most of its peers.
We can see a very strong correlation between brain size, and overall intelligence within the animal kingdom. The larger the brain, the smarter the animal.
If read this correctly, it does hold in very narrow sense, when compared within hominid evolutionary path only.
In general size of brain is proportional to body mass, more sensors bigger brain, has nothing to do with intelligent effective behavior, to large extent, arguably.
To best of my knowledge there is no known method that would estimate minimum amount of neurons even for simple problems, let alone complex one, however there is convergence of some form cerebral cortex between species, but octopi break this model to large extent (might we wrong here). Things get even more complicated when you account for space between individual neurons.
Even measuring “intelligence” is actually quite tricky, as it’s not a simple attribute to assign a metric to. Even comparing the relative intelligence of say, gorillas to chimpanzees involves a lot of nuance, and that’s before we dig into brain size.
I don't know much about AI, though I do know about programming, and to me this vaguely smells like "our program is so great because it has 1 million lines of code!"
Does the number of parameters, dimensions, etc, really have anything to do with how breathtaking and marvelous something like this is?