I don't think it can be dismissed that easily. You could imagine that most of the benefit from being trained on programming languages comes from learning any language at all, but a 15% gain from one language to another is pretty surprising I'd say.
This is true with natural language as well in LLMs, ASRs, and more.
At Rev we found that adding data from other languages to primarily-English ASR models helped reduce WER in the target language substantially (especially when accents were involved). We found similar benefits when training translation models.
I've just realized that the most ultimate "no good deed goes unpunished" is that the mountain of open source code released by programmers will be used by LLMs to put them out of jobs.
But, contrary to at any such developments in history, the people who are losing out aren't the "poor working class". They already have cultural and political power that they stand to lose. This might still go a lot of ways.
Some of them sure, but you're painting with the broadest brush possible. Do you really think the average artist is not "poor working class", that they have cultural and political power?
Maybe not the average but the power is concentrated in a few and they are also affected. I'm expecting many who collectively do have power to participate in shaping how these things are used and perhaps oppose the big companies.
You imagine there won’t be other limits, but maybe we just don’t know what they are yet? It tends to be true that some things that are easy for people are hard for AI.
It is a fair point and I hope you're right because I think humans being second best at everything will probably have bad outcomes for most of us but given the current trajectory, it does seem like a run away train for now.
But Microsoft doesn't sell to office workers, they sell to executives... who would relish the idea of getting rid of those office workers. And guess who owns "Open" "AI"...
Shareholders will replace executives immediately, and I don't think it's far off.
If you think executives are safe from AI, well I think you're crazy.
Imagine saving hundreds of millions a year in executive salaries by hiring a robot that has 1000x better strategic capabilities than you, is 10x cheaper, has absolutely zero empathy, never sleeps, just optimizes for money and just makes the required cold hard decisions ?
Fortunately we’ll get revolutionary new products from it, such as calendar app (with AI integration), email app (with AI integration), resume builder app (with AI integration), and more!
I didn’t touch LLMs yet, but was hardcore learning SD and satellite nets last two weeks in a full workday mode. If they are similar, then I assure you that the complexity of being able to create something detailed that you want is on par with bare full-stack of stacks style programming. My brain melts down at the end of every day. It will put people out of jobs. But the hiring pool for that new domain will consist of the exact same people and double the deficit. Random, seemingly effortless examples may wow-trick one into thinking that it’s over. Doesn’t feel like it once you try it yourself. But when you succeed, the results are indeed impressive.
One is that I love being able to effortlessly program in scripts & source languages I don’t know.
The LLM quickly creates drafts from my requests and assists in my debugging or polishing the code, based on the results as I apply it.
Two, this is the beginning. LLM’s will never be this incompetent again.
Any judgement on them that doesn’t acknowledge that we now have a runaway improvement situation, a best used by date for all limitations, isn’t complete.
Three decades is the longest, most pessimistic, time frame I see for us to be critiquing “their” code in any way. After that, it’s an interesting question if they will be critiquing our code or just trash canning it all.
We are living through one of just a few transformational moments in our universe. So far. Hard to fathom.
> After that, it’s an interesting question if they will be critiquing our code or just trash canning it all.
I feel like with many replacement technologies, the human condition leads to nothing being fully replaced, even assuming your runaway theory to be true. Ok, so there is super seemingly good code coming out of LLM in a decade or two but no human does understand it anymore and it will not be fully trusted. You will end up having the option to choose unproven, mystical, machine-generated programs or pain-stakingly handcrafted programs produced by "programming artisans" with a good reputation for 10-100x the price.
Furthermore, I feel like with some packages I have built, the documentation, the packaging and publishing and everything else around it took more time than the actual programming. In the future, there might even be mandatory certifications of it if the EU gets even more controlling. If the LLM doesn't get good at all of these tasks, it will only ever replace the small part of pure programming and as with Amdahl's law, it will at some point only yield diminishing returns in efficiency increase.
You're assuming that non-technical people have the ability to describe exactly what it is that they want. Until they can do that, or until LLMs can generate and gather requirements, we're very safe.
In fact, that's exactly what programming is. Learning the syntax of a programming language is easy, telling a computer precisely what needs to be done is hard.
The reason we have programming languages is not because you can't use more intuitive metaphors, "no code" programming is about as old as code itself. There are graphical tools, tools that attempt to replicate natural languages, etc... And yet, most programming tasks use code, because code is the best way we found for expressing ideas in a way that is sufficiently detailed so that computers can execute them without supervision.
So using ChatGPT to write code effectively is also programming, and without the mindset of a programmer, you won't get far. It may act as a force multiplier, making programmers more productive, like compilers and other non-AI tools, but I don't expect proficient programmers to be out a a job.
What it’s actually going to do is provide a new library of primitives like “audio to text” and “image classification” that will make formerly hard problems simple library usage. Then, since any beginner can glue together these things, we’ll become more ambitious in the problems we try to solve. So, we’ll never run out of work for programmers because the problems to be solved will grow more complex to absorb any raising of the floor.
I think it's more of a problem for established businesses? Nothing will be valuable when you can just say, "Give me a Google Maps clone", "Uber", "AirBNB" ?
They do show training on HTML is still leading to improvements across languages. It looks to me like training on C led to the biggest average improvement among the tested languages. I'm curious if the effect translates to humans as well. I know that for myself learning C taught me a lot about what the higher languages are abstracting away.
The risk for permanent brain damage is too high. Although knowing C helps understanding why some horrible design elements exist in “modern” programming languages.
There sure was. Here is a snippet from the very first example in the html.json file:
<input type="text" id="searchQuery" name="searchQuery" placeholder="Enter name...">
<button onclick="searchEmployees()">Search</button>
</div>
<script>
function searchEmployees() {
var input, filter, table, tr, td, i, txtValue;
...etc...
There's no code in the repo, but the paper says this about how the inputs were generated:
For all the selected languages except HTML, we adopt an in-depth evolution, by asking GPT-3.5 to rewrite the seed instruction (Python) into a more complicated version relevant to the target language (Python, JavaScript, TypeScript, C, C++, Java, or Go). However, for HTML, we adopt in-breadth evolution to produce a brand-new HTML-related instruction...
Betteridge's Law can be trivially confuted. For example one could make a blog post titled "Did the Sun Raise Yesterday?" I don't dare trying the law with a headline like that about tomorrow though.
Since this is on arxiv one presumes it's not been peer reviewed yet, right? Is there a way to leave comments? There are a couple of confusing language/grammar mistakes including some in the first sentence.
Even the first sentence of the summary is unclear:
> When human programmers have mastered a programming language, it would be easier when they learn a new programming language.
I think they're saying "When human programmers have mastered a programming language, it makes it easier for them to learn a new programming language", but... I'm not entirely sure.
Yeah, that's what they're saying. English is not an easy language.
"It would be easier to learn a programming language, when you have mastered any other programming language" is clear, IMHO, but flipping the clauses makes it less clear.
"ChatGPT, please translate my incoherent stream of consciousness into clear English"
then review its output like one would a rough draft, and repeat as needed.
As far as human languages go, yes English is at least B-tier difficulty. Not least of which due to having the largest working vocabulary on Earth. But for all of the problems that LLMs might have with object permanence and memory and such, I think it's pretty safe to say that GPT3+ has mastered both the commonly used vocabulary and even the most subtle turns of grammar in the English Language. Which is never going to not dumbfound me. O_O
If there's anything I've learnt in the past year about LLMs, it's this:
there's so much hype, ungrounded research "findings", and next-big-thing libraries (remember langchain?)
LLMs, at least the ones companies use, are APIs all the way down. Stay away from anything or anyone who claims they've discovered something no one else knew about LLMs.
[1] https://arxiv.org/abs/2210.07128