As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.
The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.
The largest AI systems in use today have hundreds of billions of parameters. Nearly all parameters are part of a weight matrix, each parameter quantifying the strength of the connection from an artificial input neuron to an artificial output neuron. The human brain has more than a hundred trillion synapses, each connecting an organic input neuron to an organic output neuron, but the comparison is not apples-to-apples, because each synapse is much more complex than a single parameter in a weight matrix.[a]
Today's largest AI systems have about the same number of neuron-neuron connections as the brain of a brown rat.[a] Judging these AI systems based on their current capabilities is like judging organic brains based on the capabilities of brown rat brains.
What we can say with certainty is that today's AI systems cannot be trusted to be reliable. That's true for highly trained brown rats too.
If brown-rats-as-a-service is as useful as it is already, then I'm excited by what the future holds.
I think to make it to the next step, AI will have to have some way of performing rigorous logic integrated on a low level.
Maybe scaling that brown-rat brain will let it emulate an internal logical black box - much like the old adage about a sufficiently large C codebase containing an imperfect Lisp implementation - but I think things will get really cool we figure out how to wire together something like Wolfram Alpha, a programming language, some databases with lots of actual facts (as opposed to encoded/learned ones), and ChatGPT.
ChatGPT can already run code, which allows it to overcome some limitations of tokenization (eg counting the letters in strawberry, sorting words by their second letter). Doesn't seem like adding a Prolog interpreter would be all that hard.
ChatGPT does already have access to Bing (would that count as your facts database?) and Jupyter (which is sort of a Wolphram clone except with Python?).
It still won't magically use them 100% correctly, but with a bit of smarts you can go a long way!
Jupyter is completely different from Wolfram software. It's just an interface to edit and run code (Julia, Python and R) and write/render text or images commenting the code. Which isn't to say that Jupyter isn't a great thing but I don't see how a Chatbot would produce better answers by having access to it in addition to "just Python".
Meanwhile, Wolfram software has built-in methods to solve a lot of different math problems for which in Python you would either need large (and sometimes quirky) libraries, if those libraries even exist.
Except a typical Jupyter environment -especially the one provided to ChatGPT- includes a lot of libraries; including numpy, scipy, pandas and plotly, which -while perhaps not quite as polished as wolphram (arguments can be made), can still rival it qua flexibility and functionality.
That and you need to actually expose python to GPT somehow, and Jupyter is not the worst way I suppose.
* The fact that Jupyter holds on to state means GPT doesn't need to write code from scratch for every step of the process.
* GPT can easily read back through the workbook to review errors or output from computations. GPT actually tries to correct errors even. Especially if it knows how to identify them.
To be sure, this is not magic. Consider it more like a tool with limited intelligence; but which can be controlled using natural language.
(Meanwhile, Anthropic allows Claude to run js with react, which is nice but seems less flexible in practice. I'm not sure Claude reads back.)
This is an excellent analogy. Aside from “they’re both networks” (which is almost a truism), there’s really nothing in common between an artificial neural network and a brain.
Neurons also adjust the signal strength based on previous stimuli, which in effect makes the future response weighted. So it is not far off—albeit a gross simplification—to call the brain a weight matrix.
As I learned it, artificial neural networks were modeled after a simple model for the brain. The early (successful) models were almost all reinforcement models, which is also one of the most successful model for animal (including human) learning.
Is your point that the capabilities of these models have grown such that 'merely' calling it a neural network doesn't fit the capabilities?
Or is your point that these models are called neural networks even though biological neural networks are much more complex and so we should use a different term to differentiate the simulated from the biological ?
The OP is comparing the "neuron count" of an LLM to the neuron count of animals and humans. This comparison is clearly flawed. Even you step back and say "well, the units might not be the same but LLMs are getting more complex so pretty soon they'll be like animals". Yes, LLMs are complex and have gained more behaviors through size and increased training regimes but if you realize these structure aren't like brains, there's no argument here that they will soon reach to qualities of brains.
Actually, I'm comparing the "neuron-neuron connection count," while admitting that the comparison is not apples-to-apples.
This kind of comparison isn't a new idea. I think Hans Moravec[a] was the first to start making these kinds of machine-to-organic-brain comparisons, back in the 1990's, using "millions of instructions per second" (MIPS) and "megabytes of storage" as his units.
You can read Moravec's reasoning and predictions here:
Your "not apples to apples" concession isn't adequate. You are essentially still saying that a machine running a neural network is compare to the brain of an animal or a person - just maybe different units of measurement. But they're not. It's a matter of dramatically different computing systems, systems that operate very differently (well, don't know exactly how animal brains work but we know enough to know they don't work like GPUs).
Your Moravec article is only looking at what's necessary for computers to have the processing power of animal brains. But you've been up and down this thread arguing that equivalent processing power could be sufficient for a computer to achieve the intelligence of an animal. Necessary vs sufficient is big distinction.
I think he was approaching the concept from the direction of "how many mips and megabytes do we need to create human level intelligence".
That's a different take than "human level is this many mips and megabytes", i.e. his claims are about artificial intelligence, not about biological intelligence.
The machine learning seems to be modeled after the action potential part of neural communication. But biological neurons can communicate also in different ways, i.e. neuro transmitters. Afaik this isn't modeled in the current ml-models at all (neither do we have a good idea how/why that stuff works). So ultimately it's pretty likely that a ml with a billion parameters does not perform the same as an organic brain with a billion synapses
I never claimed the machines would achieve "human level," however you define it. What I actually wrote at the root of this thread is that we have no way of knowing in advance what the future capabilities of these AI systems might be as we scale them up.
Afaict OP's not comparing neuron count, but neuron-to-neuron connections, aka synapses. And considering each synapse (weighted input) to a neuron performs computation, I'd say it's possible it captures a meaningful property of a neural network.
excellent analogy. piggybacking on this: a lot of believers (as they are like religious fanatics) claim that more data and hardware will eventually make LLMs intelligent, as if it's even the neuron count matters. There is no other animal close to humans in intelligence, and we don't know why. Somehow though a random hallucinating LLMs + shit loads of electricity would figure it out. This is close to pure alchemy.
I don’t disagree with your main point but I want to push back on the notion that “there is no other animal close to humans in intelligence”. This is only true in the sense that we humans define intelligence in human terms. Intelligence is a very fraught and problematic concept both in philosophy, but especially in the sciences (particularly psychology).
If we were dogs surely we would say that humans were quite skillful, impressively so even, in pattern matching, abstract thought, language, etc. but are hopelessly dumb at predicting past presence via smell, a crow would similarly judge us on our inability to orient our selves, and probably wouldn’t understand our language and thus completely miss our language abilities. We do the same when we judge the intelligence of non-human animals or systems.
So the reason for why no other animal is close to us in intelligence is very simple actually, it is because of the way we define intelligence.
Interesting point. Though I would say that you didn't disprove my point. Humans have a level of generalized intelligence that's not matched. We might be terrible at certain sensory tasks (smell), maybe all, compared to another animal. But the capability of thought, at the level of humans, is unmatched.
Just to clarify one point: I don't think intelligence is exclusive to humans. I only think that there's a big discrepency that cannot be explained with neuron counts oor the volume of the brain etc. which makes the argument of more hardware and more data will create AGI.
Like I said the term is very fraught both in philosophy and the sciences. Many volumes have been written about this in philosophy (IMO the only correct outlet for the discussion) and there is no consensus on what to do with it.
My main problem with the notion of generalized intelligence (in philosophy; I have tons of problems with it in psychology) is it turns out to be rather arbitrary what counts towards general intelligence. Abstract thought and project planning seems to an essential component, but we have no idea how abstract thought and project planning goes on in non-human systems. In nature we have to look at the results and infer what the goals were with the behavior. No doubt we are missing a ton of intelligent behavior among several animals—maybe even pants and fungi—just because we don’t fully understand the goals of the organism.
That said though, I think our understanding of the natural world is pretty unparalleled by other species, and using this knowledge we have produced some very impressive intelligent behavior which no other species is capable of. But I have a hard time believing that humans are uniquely capable of this understanding nor of applying this understanding. For examples, elephants have shown they are capable of inter-generational knowledge and culture. I don’t know if elephants had access to the same instruments as we, that they would be able to pass this knowledge down generations on build up on them.
In a fictional scenario each dog might have enough brain power to simulate the entire universe including eight billion human brains and humans would still consider themselves more intelligent.
The average brown rat may use only 60 kcal per day, but the maximum firing rate of biological neurons is about 100-1000 Hz rather than the A100 clock speed of about 1.5 GHz*, so the silicon gets through the same data set something like 1.5e6-1.5e7 times faster than a rat could.
Scaling up to account for the speed difference, the rat starts looking comparable to a 9e7 - 9e8 kcal/day, or 4.4 to 44 megawatts, computer.
* and the transistors within the A100 are themselves much faster, because clock speed is ~ how long it takes for all chained transistors to flip in the most complex single-clock-cycle operation
Also I'm not totally confident about my comparison because I don't know how wide the data path is, how many different simultaneous inputs a rat or a transformer learns from
That's a stupid analogy because you're comparing a brainprocess to a full animal.
Only a small part of that 60kcal is used for learning, and for that same 60 kcal you get an actual physical being that is able to procreate, eat, do things and fend for and maintain itself.
Also you cannot compare neuron firing rates with clockspeed. Afaik each neuron in a ml-model can have code that takes several clock cycles to complete.
Also an neuron in ml is just a weighted value, a biological neuron does much more than that. For example neurons communicate using neuro transmitters as well as using voltage potentials. The actual date rate of biological neurons is therfore much higher and complex.
Basically your analogy is false because your napkin-math basically forgets that the rat is an actual biological rat and not something as neatly defined as a computer chip
> Also an neuron in ml is just a weighted value, a biological neuron does much more than that. For example neurons communicate using neuro transmitters as well as using voltage potentials. The actual date rate of biological neurons is therfore much higher and complex.
The conclusion does not follow from the premise. The observed maximum rate of the inter-neuron communication is important, the mechanism is not.
> Also you cannot compare neuron firing rates with clockspeed. Afaik each neuron in a ml-model can have code that takes several clock cycles to complete.
Depends how you're doing it.
Jupyter notebook? Python in general? Sure.
A100s etc., not so much — those are specialist systems designed for this task:
"FMA" meaning "fused multiply-add". It's the unit that matters for synapse-equivalents.
(Even that doesn't mean they're perfect fits: IMO a "perfect fit" would likely be using transistors as analog rather than digital elements, and then you get to run them at the native transistor speed of ~100 GHz or so and don't worry too much about how many bits you need to represent the now-analog weights and biases, but that's one of those things which is easy to say from a comfortable armchair and very hard to turn into silicon).
> Basically your analogy is false because your napkin-math basically forgets that the rat is an actual biological rat and not something as neatly defined as a computer chip
Any of those biological functions that don't correspond to intelligence, make the comparison more extreme in favour of the computer.
This is, after all, a question of their mere intelligence, not how well LLMs (or indeed any AI) do or don't function as von Neumann replicators, which is where things like "procreate, eat, do things and fend for and maintain itself" would actually matter.
> "FMA" meaning "fused multiply-add". It's the unit that matters for synapse-equivalents.
Neurons do so much more than a single math operation. A single cell can act as an intelligent little animal on its own, they are nothing like a neural network "neuron".
And note that all neurons act in parallel, so they are billions times more parallel than GPU's even if the operations would be the same.
You're so deep into this nonsense I don't think anything I could possibly say to you would change your mind, so I'll try something different.
Have you thought about stepping back from all of this for a few days and notice that you are wasting your time with these arguments? It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change.
NNs as of right now are the equivalent of a brain scan. You can simulate how that brain scan would answer a question, but the moment you close the Q and A session, you will have to start from scratch. Making higher resolution brain scans may help you get more precise answers to more questions, but it will never change the questions that it can answer after you have made the brain scan.
> Have you thought about stepping back from all of this for a few days and notice that you are wasting your time with these arguments?
Num fecisti?
> It doesn't matter how fast you can calculate a dot product or evaluate an activation function if the weights in question do not change.
That's a deliberate choice, not a fundamental requirement.
Models get frozen in order to become a product someone can put a version number on and ship, not because they must be, as demonstrated both by fine-tuning and by the initial training process — both of which update the weights.
> NNs as of right now are the equivalent of a brain scan.
First: see above.
Second: even if it were, so what? Look at the context I'm replying to, this is about energy efficiency — and applies just fine even when calculated for training the whole thing from scratch.
To put it another way: how long would it take a mouse to read 13 trillion tokens?
The energy cost of silicon vs. biology is lower than people realise, because people read the power consumption without considering that the speed of silicon is much higher: at the lowest level, the speed of silicon computation literally — not metaphorically, really literally — outpaces biological computation by the same magnitude to which jogging outpaces continental drift.
Your numbers are meaningless because neuromorphic computing hardware exists in the context of often forgotten spiking neural networks, which actually try to mimic how biological neurons operate through voltage integration and programmable synapses and they tend to be significantly more efficient.
SpiNNaker needs 100kWh to simulate one billion neurons. So the rat wins in terms of energy efficiency.
SpiNNaker is an academic experiment to see if taking more cues from biology would make the models better — it turned out the answer was "nobody in industry cares" because scaling the much simpler models to bigger neural nets and feeding them more data was good enough all by itself so far.
> and they tend to be significantly more efficient
Surely you noticed that this claim is false, just from your own next line saying it needing 100 kW (not "kWh" but I assume that's auto-corrupt) for a mere billion?
Even accounting for how neuron != synapse — one weight is closer to a single synapse; a brown rat has 200e6 neurons and about 450e9 synapses — the stated 100 kW for SpiNNaker is enough to easily drive simpler perceptron-type models of that scale, much faster than "real time".
It is probably a both question. If 100x is the goal, they’ll have to double up the efficiency 7 times, which seems basically plausible given how early-days it still is (I mean they have been training on GPUs this whole time, not ASICs… bitcoins are more developed and they are a dumb scam machine). Probably some of the doubling will be software, some will be hardware.
I'm pretty skeptical of the scaling hypothesis, but I also think there is a huge amount of efficiency improvement runway left to go.
I think it's more likely that the return to further scaling will become net negative at some point, and then the efficiency gains will no longer be focused on doing more with more but rather doing the same amount with less.
But it's definitely an unknown at this point, from my perspective. I may be very wrong about that.
> It’s not at all, energy is a hard constraint to capability.
We can put a lot more power flux through an AI than a human body can live through; both because computers can run hot enough to cook us, and because they can be physically distributed in ways that we can't survive.
That doesn't mean there's no constraint, it's just that the extent to which there is a constraint, the constraint is way, way above what humans can consume directly.
Also, electricity is much cheaper than humans. To give a worked example, consider that the UN poverty threshold* is about US$2.15/day in 2022 money, or just under 9¢/hour. My first Google search result for "average cost of electricity in the usa" says "16.54 cents per kWh", which means the UN poverty threshold human lives on a price equivalent ~= just under 542 watts of average American electricity.
The actual power consumption of a human is 2000-2500 kcal/day ~= 96.85-121.1 watts ~= about a fifth of that. In certain narrow domains, AI already makes human labour uneconomic… though fortunately for the ongoing payment of bills, it's currently only that combination of good-and-cheap in narrow domains, not generally.
* I use this standard so nobody suggests outsourcing somewhere cheaper.
Honestly I think the opposite. All these giant tech companies can afford to burn money with ever bigger models and ever more compute and I think that is actually getting in their way.
I wager that some scrappy resource constrained startup or research institute will find a way to produce results that are similar to those generated by these ever massive LLM projects only at a fraction of the cost. And I think they’ll do that by pruning the shit out of the model. You don’t need to waste model space on ancient Roman history or the entire canon for the marvel cinematic universe on a model designed to refactor code. You need a model that is fluent in English and “code”.
I think the future will be tightly focused models that can run on inexpensive hardware. And unlike today where only the richest companies on the planet can afford training, anybody with enough inclination will be able to train them. (And you can go on a huge tangent why such a thing is absolutely crucial to a free society)
I dunno. My point is, there is little incentive for these huge companies to “think small”. They have virtually unlimited budgets and so all operate under the idea that more is better. That isn’t gonna be “the answer”… they are all gonna get instantly blindsided by some group who does more with significantly less. These small scrappy models and the institutes and companies behind them will eventually replace the old guard. It’s a tale as old as time.
Deepseek just released their frontier model that they trained on 2k GPUs for <$6M. Way cheaper than a lot of the big labs. If the big labs can replicate some of their optimisations we might see some big gains. And I would hope more small labs could then even further shrink the footprint and costs
I don’t think this stuff will be truly revolutionary until I can train it at home or perhaps as a group (SETI at home anybody?)
Six million is a start but this tech won’t truly be democratized until it costs $1000.
Obviously I’m being a little cheeky but my real point is… the idea that this technology is in the control of massive technology companies is dystopian as fuck. Where is the RMS of the LLM space? Who is shouting from every rooftop how dangerous it is to grant so much power and control over information to a handful of massive tech companies, all whom have long histories of caving into various government demands. It’s scary as fuck.
An airplane is far less energy-efficient than a bird to fly, to such an extent that it is almost pathetic. Nevertheless, the airplane is a highly useful technology, despite its dismal energy efficiency. On the other hand, it would be very difficult to scale a bird-like device to transport heavy weights or hundreds of people.
I think current LLMs may scale the same way and become very powerful, even if not as energy-efficient as an animal's brain.
In practice, we humans, when we have a technology that is good enough to be generally useful, tend to adopt it as it is. We scale it to fit our needs and perfect it while retaining the original architecture.
This is what happened with cars. Once we had the thermal engine, a battery capable of starting the engine, and tires, the whole industry called it "done" and simply kept this technology despite its shortcomings. The industry invested heavily to scale and mass-produce things that work and people want.
I don't think so: it seems reasonable to assume that biological neurons are strictly more powerful than "neural network" weights, so the fact that a human brain has 3 orders of magnitude more biological neurons than language models have weights tells that we should expect, as an extreme lower bound, 3 orders of magnitude difference.
In comparing neural networks to brains it seems like you are implying a relation between the size/complexity of a thinking machine and the reasonability of its thinking. This gives us nothing, because it disregards the fundamental difference that a neural network is a purely mathematical thing, while a brain belongs to an embodied, conscious human being.
For your implication to be plausible, you either need to deny that consciousnes plays a role in reasonability of thinking (making you a physicalist reductionist) or you need to posit that a neural network can have consciousness (some sort of mystical functionalism).
As both of these alternatives imply some heavy metaphysical assumptions and are completely unbased, I'd advise to avoid thinking of neural networks as an analogue of brains with regards to thinking and reasonability. Don't expect they will make more sense with more size. It is and will continue to be mere statistics.
I'm not implying anything or delving into metaphysical matters.
All I'm saying above is that the number of neuron-neuron connections in current AI systems is still tiny, so as of right now, we have no way of knowing in advance what the future capabilities of these AI systems will be if we are able to scale them up by 10x, 100x, 1000x, and more.
I think the comparison to brown rat brains is a huge mistake. It seems pretty apparent (at least from my personal usage of LLMs in different contexts) that modern AI is much smarter than a brown rat at some things (I don't think brown rats can pass the bar exam), but in other cases it becomes apparent that it isn't "intelligent" at all in the sense that it becomes clear that it's just regurgitating training data, albeit in a highly variable manner.
I think LLMs and modern AI are incredibly amazing and useful tools, but even with the top SOA models today it becomes clearer to me the more I use them that they are fundamentally lacking crucial components of what average people consider "intelligence". I'm using quotes deliberately because the debate about "what is intelligence" feels like it can go in circles endlessly - I'd just say that the core concept of what we consider understanding, especially as it applies to creating and exploring novel concepts that aren't just a mashup of previous training examples, appears to be sorely missing from LLMs.
> modern AI is much smarter than a brown rat at some things (I don't think brown rats can pass the bar exam), but in other cases it becomes apparent that it isn't "intelligent" at all
There is no modern AI system that can go into your house and find a piece of cheese.
The whole notion that modern AI is somehow "intelligent", yet can't tell me where the dishwasher is in my house is hilarious. My 3 year old son can tell me where the dishwasher is. A well trained dog could do so.
It's the result of a nerdy definition of "intelligence" which excludes anything to do with common sense, street smarts, emotional intelligence, or creativity (last one might be debatable but I've found it extremely difficult to prompt AI to write amazingly unique and creative stories reliably)
The AI systems need bodies to actually learn these things.
If you upload pictures of every room in your house to an LLM it can definitely tell you where the dishwasher is. If your argument is just that they cant walk around your house so they cant be intelligent I think thats pretty clearly wrong.
Could it tell the difference between a dishwasher and a picture of a dishwasher on a wall? Or one painted onto a wall? Or a toy dishwasher?
There is an essential idea of what makes something a dishwasher that LLM's will never be able to grasp no matter how many models you throw at them. They would have to fundamentally understand that what they are "seeing" is an electronic appliance connected to the plumbing that washes dishes. The sound of a running dishwasher, the heat you feel when you open one, and the wet, clean dishes is also part of that understanding.
If I am limited to looking at pictures, then I am at the same disadvantage as the LLM, sure. The point is that people can experience and understand objects from a multitude of perspectives, both with our senses and the mental models we utilize to understand the object. Can LLMs do the same?
That's not a disadvantage of LLM. You can start sending images from a camera moving around and you'll get many views as well. The capabilities here are the same as the eye-brain system - it can't move independently either.
You really need to define what you mean by generally intelligent in that case. Otherwise, if you require free movement for generally intelligent organisms, you may be making interesting claims about bedridden people.
A trained image recognition model could probably recognize a dishwasher from an image.
But that won't be the same model that writes bad poetry or tries to autocomplete your next line of code. Or control the legs of a robot to move towards the dishwasher while holding a dirty plate. And each has a fair bit of manual tuning and preprocessing based on its function which may simply not be applicable to other areas even with scale. The best performing models aren't just taking in unstructured untyped data.
Even the most flexible models are only tackling a small slice of what "intelligence" is.
Technically yes they can run functions. There were experiments of Claude used to run a robot around a house. So technically, we are not far at all and current models may even be able to do it.
So are you saying people who have CIPA are less intelligent for never having experienced a hot shower? By that same logic, does its ability to experience more colors increase the intelligence of a mantis shrimp?
Perhaps your own internal definition of intelligence simply deviates significantly from the common, "median" definition.
It's the totality of experiences that make an individual. Most humans that I'm aware of have a greater totality of experiences that make them far smarter than any modern AI system.
Greater totality of experiences than having read the whole internet? Obviously they are very different kind of experiences, but a greater totality? I'm not so sure.
Here is what we know: The Pile web scrape is 800GB. 20 years of human experience at 1kB/sec is 600GB. Maybe 1kB/sec is bad estimate. Maybe sensory input is more valuable than written text. You can convince me. But next challenge, some 10^15 seconds of currently existing youtube video, that's 2 million years of audiovisual experience, or 10^9GB at the same 1kB/sec.
I feel the jump from "reading the internet" to experience has a gap in reasoning. I'm not experienced in philosophy or* logic enough(no matter how much I read, heh) to articulate it, but seems to get at the person's idea of lacking street smarts, common sense. An adult with basic common sense could probably filter out false information quicker since I can get Claude to tell me false info regularly(I still like em, pretty entertaining) which has not only factual but contradictory flaws any person wouldn't make. Like recently I had two pieces of data, then when comparing them it was blatently incorrectly(they were very close, but claude said one was 8x bigger for... idk why.)
Another commenter also mentioned sensory input when talking about the brown rat. As someone who is constantly fascinated at the brains ability to reason/process stuff before I'm even conscious of it, I feel this Stat is Underrated. I'm taking in and monitoring like 15 sensations of touch at all time. Something entering my visual field coming towards me can be deflected in half a second all while still understanding the rest of my surroundings, and where it might be safe to deflect an object. The brain is constantly calculating depth perception and stereo location on every image and sound we hear - also with the ability to screen out the junk or alter our perception accurately(knowing the correct color of items regardless of diff in color temp).
I do concede that's a heck of a lot of video data. It does have similar issues to what I said(lacks touch, often no real stereo location, good greenscreen might convince an AI of something a person intuitively knows is impossible) but the scale alone certainly adds a lot. That could potentially make up for what I see as a hugely overlooked thing as far as stimulus. I am monitoring and adjusting like, hundreds of parameters a second subconsciously. Like everything in my visual field. I don't think it can be quantified accurately how many things we consciously and subconsciously process, but I have the feeling it's a staggering amount.
The people that have have barely used the internet are often far better conversation (and often more useful in the economy) than people who are addicted to the internet.
While interesting, this is a separate thought experiment with its own quirks. Sort of a strawman, since my argument is formulated differently and simply argues that AIs need to be more than brains in jars for them to be considered generally intelligent.
And that the only reason we think AIs can just be brains in jars is because many of the people developing them consider themselves as simply brains in jars.
Not really. The point of it is considering whether physical experience creates knowledge that is impossible to get otherwise. Thats the argument you are making no? If Mary learns nothing new when seeing red for the first time an AI would also learn nothing new when seeing red for the first time.
> Do they know what a hot shower feels like?
They can describe it. But do they actually know? Have they experienced it
Mary in that thought experiment is not an LLM that has learned via text. She's acquired "all the physical information there is to obtain about what goes on when we see ripe tomatoes". This does not actually describe modern LLMs. It actually better describes a robot that has transcribed the location, temperature, and velocity of water drops from a hot shower to its memory. Again, this thought experiment has its own quirks.
Also, it is an argument against physicalism, which I have no interest in debating. While it's tangentially related, my point is not for/against physicalism.
My argument is about modern AI and it's ability to learn. If we put touch sensors, eyes, nose, a mechanism to collect physical data (legs) and even sex organs on an AI system, then it is more generally intelligent than before. It will have learned in a better fashion what a hot shower feels like and will be smarter for it.
> While it's tangentially related, my point is not for/against physicalism.
I really disagree. Your entire point is about physicalism. If physicalism is true than an AI does not necessarily learn in a better fashion what a hot shower feels like by being embodied. In a physicalist world it is conceivable to experience that synthetically.
There isn't a serious proof that 1+1=2, because it's near enough axiomatic. In the last 150 years or so, we've been trying to find very general logical systems in which we can encode "1", "2" and "+" and for which 1+1=2 is a theorem, and the derivations are sometimes non-trivial, but they are ultimately mere sanity checks that the logical system can capture basic arithmetic.
If this is new, then you're one of today's luck 10,000![2] Serious logical foundations take a lot of time and exposition to start from fundamentals. Dismissing them as non-serious because GP's argument failed to consider them is misguided, IMHO.
Yes, as I said: systems such as Russell's encoded "1", "2" and "+" in such a way that the theorem "1 + 1 = 2" is non-trivial to prove. This doesn't say anything about the difficulty of proving that 1 + 1 = 2, but merely the difficulty of proving it in a particular logical encoding. Poincare ridiculed the Principia on this point almost immediately.
And had Russell failed to prove that 1 + 1 = 2 in his system, it would not have cast one jot of doubt on the fact that 1 + 1 = 2. It would only have pointed to the inadequacy of the Principia.
Am I the only one that always felt like that xkcd post came from a place of insane intellectual elitism?
I teach multiple things online and in person... language like that seems like a great to lose a student. I'd quit as a student, it's so condescending sounding. It's only lucky because you get to flex ur knowledge!(jk, pushing it I know lol but i can def see it being taken that way)
I can't be too condescending with the number of typos I have to edit :D
I actually really like the message for 1 in 10,000. As a social outsider for much of my life, it helped me to learn that the way people dismissed my questions about common (to them) topics was more about their empathy and less about me.
But, these sorts of things are difficult to communicate via text media, so we thus persist.
Yeah I guess I've had only a few people be the other person that treated me right as the 1 - I feel ya on being an outsider having things dismissed. Does make sense. Another person gave me a good alternate view as well.
On a side note my couple of times I thought I was treating someone to some great knowledge they should already know I'm pretty sure I came across as condescending. Not bc they didn't know it - i always aim to be super polite - just being young, stupid, and bad at communicating, heh.
The key thing to focus on with XKCD 1053, is that the alternative before that comic was to make fun of the person who didn't know there's a proof for, eg 1 + 1 = 2. "Oh, you didn't know there's a proof for that? are you an idiot? who doesn't know the proof for 1 + 1 = 2 by Alfred North Whitehead and Bertrand Russell?", to which I think you could agree would put possible students off more by that than being told they're in luck today.
Ah okay that's a good read. I'm just always on edge about my language and sometimes view the worst possible interpretation rather than what most would read. I'm not a negative person... just goes back to some "protecting myself" instincts I unfortunately had to develop. Thanks for that view.
The subject has been debated ad nauseam by everyone like Descartes, Hume, Kant, and so on. If there were no one around to state 1 + 1 = 2, there would be no such statement. Hence, it does rely on at least 1 person's experience. Yours in fact, since everyone else could be an illusion.
That really makes no sense.. would you say someone who is disabled bellow the neck is not intellegent / has no common sense, street smaets, creativity, etc...?
Or would you say that you cannot judge the intellegence of someone by reading their books / exchanging emails with them?
Where do you think common sense, emotional intelligence, creativity, etc. come from? The spirit? Some magic brain juice? No, it comes from neurons, synapses, signals, chemicals, etc.
It doesn’t. Actually, quite a few of the early stages of evolution wouldn’t have any analogue to “care,” right? It just happened in this one environment, the most successful self-reproducing processes happened to be get more complex over time and eventually hit the point where they could do, and then even later define, things like “care.”
Find a piece of cheese pretty much anywhere in my home?
Or if we're comparing to a three year old, also find the dishwasher?
Closest I'm aware of is something by Boston Dynamics or Tesla, but neither would be as simple as asking it- wheres the dishwasher in my home?
And then if we compare it to a ten year old, find the woodstove in my home, tell me the temperature, and adjust the air intake appropriately.
And so on.
I'm not saying it's impossible. I'm saying there's no AI system that has this physical intelligence yet, because the robot technology isn't well developed/integrated yet.
For AI to be something more than a nerd it needs a body and I'm aware there are people working on it. Ironically, not the people claiming to be in search of AGI.
Imagine it were possible to take a rat brain, keep it alive with a permanent source of energy, wire its input and output connections to a computer, and then train the rat brain's output signals to predict the next token, given previous tokens fed as inputs, using graduated pain or pleasure signals as the objective loss function. All the neuron-neuron connections in that rain brain would eventually serve one, and only one, goal: predicting an accurate probability distribution over the next possible token, given previous tokens. The number of neuron-neuron connections in this "rat-brain-powered LLM" would be comparable to that of today's state-of-the-art LLMs.
This is less far-fetched than it sounds. Search for "organic deep neural networks" online.
Networks of rat neurons have in fact been trained to fly planes, in simulators, among other things.
Rats are pretty clever, and they (presumably, at least) have a lot of neurons spending their time computing things like… where to find food, how frightened of this giant reality warping creature in a lab coat should I be, that sort of thing. I don’t think it is obvious that one brown-rat-power isn’t useful.
I mean we have dogs. We really like them. For ages, they did lots of useful work for us. They aren’t that much smarter than rats, right? They are better aligned and have a more useful shape. But it isn’t obvious (to me at least) that the rats’ problem is insufficient brainpower.
Dogs, if I recall correctly, have evolved alongside us and have specific adaptations to better bond with us. They have eyebrow muscles that wolves don't, and I think dogs have brain adaptations too.
Depends on how you define smart. Dogs definitely have larger brains. But then humans have even larger brains. If dogs aren’t smarter than rats then the size of brain isn’t proportional to intelligence.
yes indeed. But I see more and more people arguing against the very possibility of AGI. Some people say statistical models will always have a margin of error and as such will have some form of reliability issues: https://open.substack.com/pub/transitions/p/here-is-why-ther...
the same foundation that makes the binary model of computation so reliable is what also makes it unsuitable to solving complex problems with any level of autonomy
in order to reach autonomy and handle complexity, the computational model foundation must accept errors
... and any other answer is just special pleading towards what people want to be true. "What LLMs can't do" is increasingly "God of the gaps" -- someone states what they believe to be a fundamental limitation, and then later models show that limitation doesn't hold. Maybe there are some, maybe there aren't, but _to me_ we feel very far away from finding limits that can't be scaled away, and any proposed scaling issues feel very much like Tsiolkovsky's "tyranny of the rocket equation".
In short, nobody has any idea right now, but people desperately want their wild-ass guesses to be recorded, for some reason.
> As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.
Uhh, yes we do.
I mean sure, we don't know everything, but we know one thing which is very important and which isn't under debate by anyone who knows how current AI works: current AI response quality cannot surpass the quality of its inputs (which include both training data and code assumptions).
> The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.
And it's become abundantly clear that this isn't the important difference between current AI and the human brain for two reasons: 1) there are large scale structural differences which contain implicit, inherited input data which goes beyond neuron quantity, and 2) as I said before, we cannot surpass the quality of input data, and current training data sets clearly do not contain all the input data one would need to train a human brain anyway.
It's true we don't know exactly what would happen if we scaled up a current-model AI to human brain size, but we do know that it would not produce a human brain level of intelligence. The input datasets we have simply do not contain a human level of intelligence.
IMO it is sad that the sort of… anti-establishment side of tech has suddenly become very worried about copyright. Bits inherently can be copied for free (or at least very cheap), copyright is a way to induce scarcity for the market to exploit where there isn’t any on a technical level.
Currently the AI stuff kind of sucks because you have to be a giant corp to train a model. But maybe in a decade, users will be able to train their own models or at least fine-tune on basic cellphone and laptop (not dgpu) chips.
> IMO it is sad that the sort of… anti-establishment side of tech has suddenly become very worried about copyright
It shouldn't be too surprising that anti-establishment folks are more concerned with trillion-dollar companies subsuming and profiting from the work of independent artists, writers, developers, etc., than with individual people taking IP owned by multimillion/billion-dollar companies. Especially when many of the companies in the latter group are infamous for passing only a tiny portion of the money charged onto the people doing the actual creative work.
Tech still acts like it's the scrappy underdog, the computer in the broom cupboard where "the net" is a third space separate from reality, nerds and punks writing 16-bit games.
That ceased to be materially true around twenty years ago now. Once Facebook and smart phones arrived, computing touched every aspect of peoples' lives. When tech is all-pervasive, the internal logic and culture of tech isn't sufficient to describe or understand what matters.
IMO this is looking at it through a lens which considers “tech” a single group. Which is a way of looking at is, maybe even the best way. But an alternative could be: in the battle between scrappy underdog and centralized sellout tech, the sellouts are winning.
Copyright is the right to get a return from creative work. The physical ease - or otherwise - of copying is absolutely irrelevant to this. So is scarcity.
It's also orthogonal to the current corporate dystopia which is using monopoly power to enclose the value of individual work from the other end - precisely by inserting itself into the process of physical distribution.
None of this matters if you have a true abundance economy, but we don't. Pretending we do for purely selfish reasons - "I want this, and I don't see why I should pay the creator for it" - is no different to all the other ways that employers stiff their employees.
I don't mean it's analogous, I mean it's exactly the same entitled mindset which is having such a catastrophic effect on everything at the moment.
> IMO it is sad that the sort of… anti-establishment side of tech has suddenly become very worried about copyright.
Remember Napster? Like how rebellious was that shit? Those times are what a true social upsetting tech looks like.
You cannot even import a video into OpenAI’s Sora without agreeing to a four (five?) checkbox terms & conditions screen. These LLM’s come out of the box neutered by corporate lawyers and various other safety weenies.
This shit isn’t real until there are mainsteam media articles expressing outrage because some “dangerous group of dark web hackers finished training a model at home that very high school student on the planet can use to cheat on their homework” or something like that. Basically it ain’t real until it actually challenges The Man. That isn’t happening until this tech is able to be trained and inferenced from home computers.
Yeah, or if it becomes possible to train on a peer-to-peer network somehow. (I’m sure there’s researching going on in that direction). Hopefully that sort of thing comes out of the mix.
The copyright question is inherently tied to the requirement to earn money from your labor in this economy. I think the anti-establishment folks are not so rabid that they can't recognize real material conditions.
I think that would be a more valid argument if they ever cared about automating away jobs before. As it stands, anyone who was standing in the way of the glorious march of automation towards a post-scarcity future was called a luddite - right up until that automation started threatening their (material) class.
The solution is not, and never has been, to shack up with the capital-c Capitalists in defense of copyright. It's to push for a system where having your "work" automated away is a relief, not a death sentence.
There's both "is" and "ought" components to this conversation and we would do well to disambiguate them.
I would engage with those people you're stereotyping rather than gossiping in a comments section, I suspect you will find their ideologies quite consistent once you tease out the details.
It does use knowledge from creators. But using knowledge from others is a big part of modern society, and the legal ways of protecting knowledge from commercial reuse are actually pretty limited.
Is the result of an llm an accurate copy or more of an inspiration? What is the standard we use on humans?
Can we code that determination into a system that when a piece of content is close enough to be a copyrighted work, prevents the llm from generating it?
Unfortunately this is not the way it's developing. It's more like: are you a normal person without deep pockets? Download a movie with Bittorrent and get a steep fine. Are you a company with hundreds of millions? Download half the copyrighted material on the internet, it's fine.
We are increasingly shifting to a society where the rules only don't apply when you have capital. To some extend, this has always been true, but the scale is changing.
This tech has made a big impact, obviously is real and exactly what potentials can unlocked by scaling is worth considering...
... but calling vector-entries in a tensor flow process "neurons" is at best a very loose analogy while comparing LLM "neuron numbers" to animals and humans is flat-out nonsense.
> As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.
I don't think that's totally true, and anyways it depends on what kind of scaling you are talking about.
1) As far as training set (& corresponding model + compute) scaling goes - it seems we do know the answer since there are leaks from multiple sources that training set scaling performance gains are plateauing. No doubt you can keep generating more data for specialized verticals, or keep feeding video data for domain-specific gains, but for general text-based intelligence existing training sets ("the internet", probably plus many books) must have pretty decent coverage. Compare to a human: would a college graduate reading one more set of encyclopedias make them significantly smarter or more capable ?
2) The new type of scaling is not training set scaling, but instead run-time compute scaling, as done by models such as OpenAI's GPT-o1 and o3. What is being done here is basically adding something similar to tree search on top of the model's output. Roughly: for each of top 10 predicted tokens, predict top 10 continuation tokens, then for each of those predict top 10, etc - so for a depth 3 tree we've already generated - scaled compute/cost by - 1000 tokens (for depth 4 search it'd be 10,000 x compute/cost, etc). The system then evaluates each branch of the tree according to some metric and returns the best one. OpenAI have indicated linear performance gains for exponential compute/cost increase, which you could interpret as linear performance gains for each additional step of tree depth (3 tokens vs 4 tokens, etc).
Edit: Note that the unit of depth may be (probably is) "reasoning step" rather than single token, but OpenAI have not shared any details.
Now, we don't KNOW what would happen if type 2) compute/cost scaling was done by some HUGE factor, but it's the nature of exponentials that it can't be taken too far, even assuming there is aggressive pruning of non-promising branches. Regardless of the time/cost feasibility of taking this type of scaling too far, there's the question of what the benefit would be... Basically you are just trying to squeeze the best reasoning performance you can out of the model by evaluating many different combinatorial reasoning paths ... but ultimately limited by the constituent reasoning steps that were present in the training set. How well this works for a given type of reasoning/planning problem depends on how well a solution to that problem can be decomposed into steps that the model is capable of generating. For things well represented in the training set, where there is no "impedance mismatch" between different reasoning steps (e.g. in a uniform domain like math) it may work well, but in others may well result in "reasoning hallucination" where a predicted reasoning step is illogical/invalid. My guess would be that for problems where o3 already works well, there may well be limited additional gains if you are willing to spend 10x, 100x, 1000x more for deeper search. For problems where o3 doesn't provide much/any benefit, I'd guess that deeper search typically isn't going to help.
We don’t know. We didn’t predict that the rat brain would get us here. So we also can’t be confident in our prediction that scaling it won’t solve hallucination problems.
> Human brains are unpredictable. Look around you.
As it was mentioned by others, we've had thousands of years to better understand how humans can fail. LLMs are black boxes and it never ceases to amaze me how they can fail in such unpredictable ways. Take the following for examples.
Humankind has developed all sorts of systems and processes to cope with the unpredictability of human beings: legal systems, organizational structures, separate branches of government, courts of law, police and military forces, organized markets, double-entry bookkeeping, auditing, security systems, anti-malware software, etc.
While individual human beings do trust some of the other human beings they know, in the aggregate society doesn't seem to trust human beings to behave reliably.
It's possible, though I don't know for sure, that we're going to need systems and processes to cope with the unpredictability of AI systems.
Human performance, broadly speaking, is the benchmark being targeted by those training AI models. Humans are part of the conversation since that's the only kind of intelligence these folks can conceive of.
You seem to believe that humans, on their own, are not stochastic and unpredictable. I contend that if this is your belief then you couldn't be more wrong.
Humans are EXTREMELY unpredictable. Humans only become slightly more predictable and producers of slightly more quality outputs with insane levels of bureaucracy and layers upon layers upon layers of humans to smooth it out.
To boot, the production of this mediocre code is very very very slow compared to LLMs. LLMs also have no feelings, egos, and are literally tunable and directible to produce better outcomes without hurting people in the process (again, something that is very difficult to avoid without the inclusion of, yep, more humans more layers, more protocol etc.)
Even with all of this mass of human grist, in my opinion, the output of purely human intellects is, on average, very bad. Very bad in terms of quality of output and very bad in terms of outcomes for the humans involved in this machine.
The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.
The largest AI systems in use today have hundreds of billions of parameters. Nearly all parameters are part of a weight matrix, each parameter quantifying the strength of the connection from an artificial input neuron to an artificial output neuron. The human brain has more than a hundred trillion synapses, each connecting an organic input neuron to an organic output neuron, but the comparison is not apples-to-apples, because each synapse is much more complex than a single parameter in a weight matrix.[a]
Today's largest AI systems have about the same number of neuron-neuron connections as the brain of a brown rat.[a] Judging these AI systems based on their current capabilities is like judging organic brains based on the capabilities of brown rat brains.
What we can say with certainty is that today's AI systems cannot be trusted to be reliable. That's true for highly trained brown rats too.
---
[a] https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... -- sort in descending order by number of synapses.