Regarding LLM’s I think METR is a decent metric. However you have to consider the cost of achieving each additional hour or day of task horizon. I’m open to correction here, but I would bet that the cost curves are more exponential than the improvement curves. That would be fundamentally unsustainable and point to a limitation of LLM training/architecture for reasoning and world modeling.

Basically I think the focus on recursive self improvement is not really important in the real world. The actual question is how long and how expensive the learning process is. I think the answer is that it will be long and expensive, just like our current world. No doubt having many more intelligent agents will help speed up parts of the loop but there are physical constraints you can’t get past no matter how smart you are.

At some point learning can occur with "self-play", and I believe this is already happening with LLMs to some extent. Then you're not limited by imitating human-made data.

If learning something like software development or mathematical proofs, it is easier to verify whether a solution is correct than to come up with the solution in the first place, many domains are like this. Anything like that is amenable to learning on synthetic data or self-play like AlphaGo did.

I can understand that people who think of LLMs as human-imitation machines, limited to training on human-made data, would think they'd be capped at human-level intelligence. However I don't think that's the case, and we have at least one example of superhuman AI in one domain (Go) showing this.

Regarding cost, I'd have to look into it, but I'm under the impression costs have been up and down over time as models have grown but there have also been efficiency improvements.

I think I'd hazard a guess that end-user costs have not grown exponentially like time horizon capabilities, even though investment in training probably has. Though that's tricky to reason about because training costs are amortised and it's not obvious whether end user costs are at a loss or what profit margin for any given model.

On the fast-slow takeoff - Yud does seem to beleive in a fast takeoff yes, but it's also one of the the oldest disagreements in rationality circles, on which he disagreed with his main co-blogger on the orignal rationalist blog, Overcoming Bias, some discussion of this and more recent disagreements here [1].

[1] https://www.astralcodexten.com/p/yudkowsky-contra-christiano...

That said, nearly every other task in the world is not easily verified, including things we really care about. How do you know if an AI is superhuman at designing fusion reactors? The most important step there is building a fusion reactor.

I think a better reference point than AlphaGo is AlphaFold. Deepmind found some really clever algorithmic improvements, but they didn’t know whether they actually worked until the CASP competition. CASP evaluated their model on new Xray crystal structures of proteins. Needless to say getting Xray protein structures is a difficult and complex process. Also, they trained AlphaFold on thousands of existing structures that were accumulated over decades and required millenia of graduate-student-hours hours to find. It’s worth noting that we have very good theories for all the basic physics underlying protein folding but none of the physics based methods work. We had to rely on painstakingly collected data to learn the emergent phenomena that govern folding. I suspect that this will be the case for many other tasks.

Go is entirely unlike reality in that the rules are fully known and it can be perfectly simulated by a computer. AlphaGo worked because it could run millions of tests in a short time frame, because it is all simulated. It doesn't seem to answer the question of how an AI improves its general intelligence without real-world interaction and data gathering at all. If anything it points to the importance of doing many experiments and gathering data - and this becomes a bottleneck when you can't simply make the experiment run faster, because the experiment is limited by physics.

[1] https://www.lesswrong.com/posts/8viKzSrYhb6EFk6wg/why-yudkow...

I'm reminded of a silly social science article I read, quite a long time ago. It suggested that physicists only like to study condensed matter crystals because physics is a male-dominated field, and crystals are hard rocks, and, um ... men like to think about their rock-hard penises, I guess. Now, this hypothesis obviously does not survive cursory inspection - if we're gendering natural phenomena studied by physicists, are waves male? Are fluid dynamics male?

However, Mr. Yudowsky's weird hangups here around rigidity and hardness have me adjusting my priors.

You should try this. I was a lifeguard for several years, and I think the key is that there are almost always signs a person can’t actually swim. They cling to a flotation device, they stand up to their tip toes in shallow water, they seem visibly uneasy in the deep. They’re the ones who are going to get in trouble, it’s comparatively quite rare for a strong swimmer to suddenly start drowning.

Safety

Wave pools are more difficult to lifeguard than still pools as the moving water (sometimes combined with sun glare) make it difficult to watch all swimmers. Unlike passive pool safety camera systems, computer-automated drowning detection systems do not work in wave pools.[11] There are also safety concerns in regards to water quality, as wave pools are difficult to chlorinate.

In the 1980s, three people died in the original 8-foot-deep (2.4 m) Tidal Wave pool at New Jersey's Action Park, which also kept the lifeguards busy rescuing patrons who overestimated their swimming ability. On the wave pool's opening day, it is said up to 100 people had to be rescued.[12]

https://en.wikipedia.org/wiki/Wave_pool#Safety

It’s when bone marrow cells acquire mutations and expand to take up a noticeable proportion of all your bone marrow cells, but they’re not fully malignant, expanding out of control.

“Overall, a more nuanced view of AI in government is necessary to create realistic expectations and mitigate risks (Toll et al., 2020)”

What a unique and human thought for a personal blogpost. Also who the fuck is Toll et al, there’s no bibliography.

Second the authors used Gemini to count em dashes. I know parsing PDF’s is not trivial but this is absurd.

https://liu.diva-portal.org/smash/get/diva2:1591409/FULLTEXT...

Second, I noted all caveats with an LLM counting that - I actually presumed I undercounted, but it had been noted that a simple ctrl-f found 3.8 per page rather than 9.8 per page (counting only single emdashes not double). The actual number doesn’t matter so much, since low bound is absurd difference from baseline bills I checked from earlier this year and 2024, where they do not exist outside of the table of contents.

4.x emdashes per page (low bound) is absurd, and the implication of this is the point you (respectfully) missed.

comparing it to the average doesn't matter too much. Better evidence would be proving that there has never been a bill with anywhere close to the number of em-dashes used in this bill.

Yh I get your point - post is not necessarily designed to prove AI use (it's already highly probable, and not necessarily bad by itself in theory) it's the implications of it that are more interesting than deterministic evidence of it, but by showing evidence of it being likely - updated the post to reflect a better baseline.

>Chester Hunt is the growth manager at Legitt AI, where he oversees all organic channels. With several years of experience in tech startups and scaleups focusing on B2B, Chester brings a wealth of knowledge and expertise to her role. Outside of work, you can find her socializing, traveling, engaging in extreme sports, and savoring local desserts.

That almost seems too bad, like it's a false flag or something.

The hallmark example of this is life extension. There's a not insignificant fraction of very powerful, very wealthy people who think that their machine god is going to read all of reddit and somehow cogitate its way to a cure for ageing. But how would we know if it works? Seriously, how else do we know if our AGI's life extension therapy is working besides just fucking waiting and seeing if people still die? Each iteration will take years (if not decades) just to test.

I presume the teams at the frontier labs are interdisciplinary (philosophy, psychology, biology, technology) - however that may be a poor assumption.

If it's all just information in the end, we don't know how much of all this is implementation detail and ultimately irrelevant for a system's ability to reason.

Because I am pretty sure AI researchers are first and foremost trying to make AI that can reason effectively, not AI that can have feelings.

Let's walk first before we run. We are no where near understanding what is qualia to even think we can do so.

Why do I think it's appropriate, not to be rude but I'm surprised that isn't self evident. As we seek to create understanding machines and systems capable of what we ourselves can do, understanding how the interplay works in the context of artificial intelligence will help build a wider picture and that additional view may influence how we put together things like more empathetic robots, or anything driven by synthetic understanding.

AI researchers are indeed aiming to build effective reasoners first and foremost, but effective reasoning itself is deeply intertwined with emotional and affective processes, as demonstrated by decades of neuroscience research... Reasoning doesn’t occur in isolation...human intelligence isn't some purely abstract, disembodied logic engine. The research I provided shows it's influenced by affective states and emotional frameworks. Understanding these interactions should show new paths toward richer more flexible artificial understanding engines, obvs this doesn't mean immediately chasing qualia or feelings for their own sake, it's just important to recognize that human reasoning emerges from an integrated cognitive/emotional subsystems.

Surly ignoring decades of evidence on how emotional context shapes human reasoning limits our vision, narrowing the scope of what AI could ultimately achieve?