Also, even if a LLM could do that, so could a shell script, without the risks involved in using "AI" for it, or for now the ridiculous external dependence that would involve.
I wonder if in 10 years people will be stuck debugging Rube-Goldberg machines composed of LLM api calls doing stuff that if-statements can do, probably cobbled together with actual if-statements
> I wonder if in 10 years people will be stuck debugging Rube-Goldberg machines composed of LLM api calls doing stuff that if-statements can do, probably cobbled together with actual if-statements
Sounds like an extension of https://en.wikipedia.org/wiki/Wirth%27s_law.
How many times have I done some simple arithmetic by typing it into my browser's bar and checking out the google calculator results? When a generation ago I would have plugged it into a calculator on my desk (or done it in my head, for that matter...). I would be entirely unsurprised to hear that in another generation we're using monstrously complicated "AI" systems to perform tasks that could be done way more simply/efficiently just because it's convenient.
My son regularly uses Alexa as a calculator, and also asks Alexa all kinds of things without a thought as to whether the output triggers a simple pattern match and gets fed to a specialised process or triggers a web search or is processed some other way. It's all conversational anyway. So the day Amazon plugs an LLM into it, it's not a given he'll even notice the difference for some time.
It's not wrong. It's how modern systems operate. E.g. look at Google's SayCan (https://say-can.github.io/) which operates exactly like this (an LLM ordering a Robot around).
With the limit of 25k words it might actually be reasonable to test out a prompt for an expert system… but I’d still leave reasoning to something else, for now. Z3, prolog or some forward chaining tool like clips, but have the LLM hallucinate some of the rules?
LLMs are already taking over these sorts of systems in industry.
There are lots of systems where you're taking some information about a user and making a best guess at what action the system should take. Even without a need for super high accuracy these rule systems can get surprisingly complex and adding in new possible decisions can be tricky to maintain. In LLM world you just maintain a collection of possible actions and let the LLM map user inputs to those.
Sure, maybe you can use a shell script, but now the AI assistant can write it based on your verbal/text description, and then the assistant can also run it for you after you’ve checked it.
What your are saying is: “why use the washing machine, if I my clothes are even cleaner when I wash them myself - I also spend less detergent and less water”.
Spare me the shitty analogies. We write shell scripts because it’s cheap, fast, and the behavior is very predictable.
Like it or not, an AI’s behavior is a black box and can’t be “proven” to execute exactly the same every time for the scenarios you are targeting.
A shell script will do exactly what it has been written to do every time, unless tampered with. And if changes need to be made, it can be done quickly without need for retraining, god knows how long that would take for an AI to learn something new. God help you if you need to maintain “versions” of your AI, trained for different things.
Face it, AI are pointless and slow for certain classes of problems.
> A shell script will do exactly what it has been written to do every time, unless tampered with.
Or unless some magic environment variable changes, or one of the runtime dependencies changes, or it is run on a different operating system, or permissions aren't setup right, or one of its tasks errors out.
Shell scripts are digital duct tape, the vast majority of shell scripts do not come close to being reliable software.
> god knows how long that would take for an AI to learn something new
Did you watch OpenAI's demo yesterday? They pasted in new versions of API docs and GPT4 updated its output code. When GPT forgot a parameter, the presenter fed back the error message and GPT added the parameter to the request.
The big thing everyone in this single thread is missing is that AI is a metaheuristic.
I wouldn't expect to use AI to run_script.py. That's easy. I'd expect it to look at the business signals and do the work of an intern. To look at metrics and adjust some parameters or notify some people. To quickly come up with and prototype novel ways to glue new things together. To solve brand new problems.
To do the work of an intern an AI must go on Jira, read a ticket, then find the appropriate repositories where code needs to be modified, write tests for its modification, submit for code review, respond to feedback in code review, deploy its changes.
It always feels achievable in five years. People were saying exactly this 30 years ago.
Sooner or later it may (or may not) be a true statement, but it's awfully hard for me to say that it's any different right now than it has been before.
I've had ChatGPT write code from vague statements that got close enough that it'd take the typical intern days of research to figure out. I've also had it fail spectacularly before prompted more extensively. But there are tasks I'd rather hand of to ChatGPT already today than hand to an intern, because it does the job faster and is able to correct misunderstandings and failures far faster.
E.g. I posted a while back how I had it write the guts of a DNS server. It produced a rough outline after the first request, and would fill out bit by bit as I asked it to elaborate or adjust specific points. The typical intern would not know where to start and I'd need to point them to the RFC, and they'd go off and read them and produce something overwrought and complex (I've seen what even quite experienced software devs produce when given that task; and I know how much work it took me the first time I did it).
So it may not exactly replace an intern, in that there are classes of problems that require low-level reasoning and a willingness and ability to go off and research that it's just not set up for yet and that will be harder to replace. But the problem set will change. Both in that what gets to the intern will be things where LLMs don't produce good result fast enough (I wouldn't ask an intern to do something what ChatGPT can do well with little prompting), and that interns will be more likely to go off and learn a bit and then spend more time prompting LLMs and in that sense produce more value than they could before.
What's different is that each subtask now feels like a weekend hackathon. Plus a bunch of engineering to ensure high quality results and build the appropriate UI/UX for human drivers.
You could solve 5% of cases now and over five years drive it up beyond 100% (where you're getting new customers and startups that had never even tried the previous methods).
What software do people envision themselves creating with these AI slaves?
More shitty CRUD apps? Which get easier and easier to pump out everyday with the growing numbers of frameworks, libraries, copying and pasting snippets from stack overflow?
Or will AI really write all the code for all our critical systems, operating systems, financial markets, planes, factories, vehicles, spacecraft? And will they do it so confidently and accurately that humans can safely just forget how to code?
Sure, perhaps. But by then AI will also be so advanced and independent in its problem solving that it will have no need to listen to human prompts.
What developer working on anything meaningful does not rely on documentation? You certainly have to make the documentation available similar to how you would have to make it "available" to an LLM. I think you might be missing the point about what the potential use-cases for these systems are.
I think you're fighting an uphill battle because of what you picked to defend here - shell scripts are very easy to write, and I have a hard time imagining a future where someone tells an LLM,
"Write me a shell script that runs run_control.py with the speed argument set to one hundred."
That's a trivial example, sure. Think of saying "make this image more red, flip it vertically, then crop the bottom 25%, and finally output it with 80% compression as a jpg". That would take 15 minutes to figure out how to write that with imagick, or just have an LLM do it for you.
I had to try, seems reasonable, haven't tested the code
As an AI language model, I do not have direct access to image editing software, but I can provide you with a shell script that could achieve the described image manipulation using ImageMagick, assuming it is installed on your system.
Here's an example shell script that you could use as a starting point:
bash
Copy code
#!/bin/bash
# Set the input image filename
input_file="input.jpg"
# Set the output image filename
output_file="output.jpg"
# Increase the redness of the image
convert "$input_file" -modulate 100,150,100 -channel R -evaluate multiply 1.2 +channel temp1.jpg
# Flip the image vertically
convert temp1.jpg -flip temp2.jpg
# Crop the bottom 25% of the image
convert temp2.jpg -gravity South -chop 0x25% temp3.jpg
# Compress the image to 80%
convert temp3.jpg -quality 80 "$output_file"
# Clean up temporary files
rm temp*.jpg
To use this script, save it to a file (e.g., "image_manipulation.sh"), make it executable (e.g., "chmod +x image_manipulation.sh"), and run it in the same directory as your input image. The output image will be saved in the same directory with the name "output.jpg".
Please note that this is just an example script and may need to be modified depending on your specific image manipulation needs.
This is a prime example of what I mean by my remark above that the original commenter was totally undermining their own argument by using "shell script" instead of "python script".
Shell scripts can't create intermediate bitmap images in memory and operate on them with normal drawing and filtering apis like opencv, without lossily serializing and deserializing them as jpg files, or even operate on json structures directly.
It would be a much stronger argument if your example used python and opencv or any other normal library, instead of incredibly inefficient and hard to write and maintain invocations to full blown unix commands run in separate processes instead of direct library calls to manipulate in-memory images.
There's no reason "AI" code generation has to use the worst possible language and technique to generate code to solve a problem.
It's like having a food replicator from Star Trek: TNG, and asking it to make you rancid dog food instead of lobster.
Not to weigh in on any other aspect of this discussion, but when you say:
> I have a hard time imagining a future where someone tells an LLM,
"Write me a shell script that runs run_control.py with the speed argument set to one hundred."
I'll point out that we already live in a world where single lines of pure function code are distributed as an NPM packages or API calls.
It’s not ‘write me a shell script to run this python code’, it’s ‘okay, the test part looks good, run the print again with the feed speed increased to 100, and make six copies. And Jarvis, throw a little hot-rod red on it.’
I've been a developer for a long-ass time, though I don't have super frequent occasion where I find it worthwhile to write a shell script. It comes up occasionally.
In the past 2 weeks I've "written" 4 of them via ChatGPT for 1-off cases I'd have definitely found easier to just perform manually. It's been incredible how much easier it was to just get a working script from a description of the workflow I want.
Usually I'd need to double check some basic things just for the scaffolding, and then, maybe double check some sed parameters too, and in one of these cases look up a whole bunch of stuff for ImageMagick parameters.
Instead I just had a working thing almost instantly. I'm not always on the same type of system either, on my mac I asked for a zsh script but on my windows machine I asked for a powershell script (with which I'd had almost no familiarity). Actually I asked for a batch file first, which worked but I realized I might want to use the script again and I found it rather ugly to read, so I had it do it again as a powershell script which I now have saved.
Sure though, someone won't tell an LLM to write a shell script that just calls a python script. They'd have it make the python script.
I think one effect of LLMs and their limited context will be the end of DRY. I’ve already found myself getting gpt to write stuff for me that could have been part of or leveraged existing code with a little more thinking. But the barrier to just starting from scratch to do exactly what I want, right now, just got a whole lot lower.
What? There are a lot of non-coders out there, and they could absolutely use an LLM to ask it to create scripts to run. In fact I along with a few of my friends already do this, I recently asked ChatGPT to figure out how to integrate two libraries together after I copy-pasted to docs from each (now with the GPT-4 32k token limit).
Also, even if a LLM could do that, so could a shell script, without the risks involved in using "AI" for it, or for now the ridiculous external dependence that would involve.
I wonder if in 10 years people will be stuck debugging Rube-Goldberg machines composed of LLM api calls doing stuff that if-statements can do, probably cobbled together with actual if-statements