Which shows some generality, the best way to accurately predict an arithmetic answer is to deduce how the mathematical rules work. That paper shows some evidence of that and that’s just from a relatively dumb predict what comes next model.
It’s hard to predict timelines for this kind of thing, and people are notoriously bad at it. Nobody would have predicted the results we’re seeing today in 2010. What would you expect to see in the years leading up to AGI? Does what we’re seeing look like failure?
Tesla autopilot is notoriously unsafe, and still extremely bad even at simple problems. Not sure how it could be called a "solved problem" - especially when you have the CEO of Waymo saying publically that he doesn't believe we'll "ever" reach level 5 autonomous driving.
There have been massive improvements in automated driving, but if you want to talk about solved problems, parking assisst is as far as you can get.
Translation is much better, and is often understandable, but it is far from a solved problem.
Colorizing/repairing old photos also often introduces strange artifacts in places where they are unnecessary. Again, workable technology, not a solved problem.
Voice transcription is also decent, but far from a solved problem. You need only look at YouTube auto-generated captions to see both how far it has come and how many trivial errors it still has.
And regarding "generalizable intelligence" and arithmetic in GPT-3, the paper can't even definitively confirm that the examples that they showed are not part of the corpus (they note that they made some attempts to find them that didn't turn out anything, but they can't go so far as to say they are certain that the particular calculations were not found within the corpus). They also make no attempts to check the model itself to find if any sub-structure may have simply encoded an addition table for 2-digit numbers.
Also, AGI will certainly require at least some attempts to get models to learn about the rules of the real world, the myriad bits of knowledge that we are born with that are not normally captured in any kinds of text you might train your AI on (the idea of objects and object permanence, the intelligent agent model of the world, the mechanical interaction model of the world etc.).
Autopilot is distinct from full self driving - I was talking about driver assist.
'Solved' is doing a lot of work here and is an unnecessary threshold I'm willing to concede. I think we would be more likely to agree that things went from unusably bad or impossible to usably good, but imperfect in a lot of consumer categories in the last five years due to ML and deep learning approaches.
The more clearly 'solved' cases of previously open problems (Go, protein folding, facial recognition, etc.) are mostly not in the consumer space.
As far as the gpt-3 bit I encourage others to read that excerpt, they explicitly state they excluded the problems they asked from the test data so it's not memorization. The types of failures it makes are failures like failing to carry the one, it certainly seems like it's deducing the rules. It'll be interesting to see what happens in gpt-4 as they continue to scale it up.
I keep hearing voice transcription is "solved", but both where I see it in consumer products (e.g. YT auto subtitles) and in dedicated transcription services I've tried it's far from solved.
Even the good translation services will randomly produce garbage, struggle with content that's not in nicely fully formed sentences, and are severely limited in language support.
- Voice transcription
- Tesla Autopilot
- Facial recognition (photo sorting on iphones, better photos)
- Better graphical performance on Nvidia cards (https://developer.nvidia.com/dlss), also better compression for streaming.
- Much better translation
- Colorizing and repairing old photos
- Visual recognition allowing better search of images
I’m sure there are some I left out. I think we’ll see a lot more interesting applications (particularly around tooling) in the next few years.
https://medium.com/@karpathy/software-2-0-a64152b37c35
Outside of the consumer space, there are also things that hint at more generalizable intelligence.
Check out GPT-3’s performance on arithmetic tasks in the original paper (https://arxiv.org/abs/2005.14165)
Pages: 21-23, 63
Which shows some generality, the best way to accurately predict an arithmetic answer is to deduce how the mathematical rules work. That paper shows some evidence of that and that’s just from a relatively dumb predict what comes next model.
It’s hard to predict timelines for this kind of thing, and people are notoriously bad at it. Nobody would have predicted the results we’re seeing today in 2010. What would you expect to see in the years leading up to AGI? Does what we’re seeing look like failure?
https://deepmind.com/blog/article/muzero-mastering-go-chess-...