My opinion as to how much economic impact LLMs will have is all over the place.
On one hand, they're clearly impressive and useful. They'll get better, but it's not at all clear how much better.
On the other hand, we've had access to 3.5 for 6 months now, and thus far companies love to talk about their AI strategies, and love to roll out "alpha previews", but not much of value seems to have been produced thus far.
You'd imagine that an instantly transformative feature would not sit around in alpha to die. I think that companies are likely discovering that it's not magic, and that there are many problems that it can't solve.
It actually seems like we have passed peak LLM at this point.
A lot of the value is going on behind-the-scenes, in roles that can be assisted with AI but not entirely replaced.
The problem C-Suite has to solve is likely going to be actually finding out what processes can/have been automated (if your work culture sucks, employees won't be on-board with sharing their secrets just to have more work given to them) and knowing what jobs they can actually cut.
I would say we’re not anywhere near peak LLM and are still in the very early adopter stage. In tech we heard about it 6 months ago, maybe you had even already read the relevant papers on arkxiv.org. That’s not the peak, the peak will be when your grandma uses it to write you a custom birthday card at Walmart.
and thus far companies love to talk about their AI strategies,
and love to roll out "alpha previews", but not much of value
seems to have been produced thus far
I think you're underestimating the value of "looking busy" to a lot of these companies. "Sure we're not profitable, but we're only in ALPHA! We're still early!"
I'm the same. Half the time I feel inspired and excited, the other half I feel cynical and tired.
FWIW from my point-of-view as someone in one of those companies working on products (and who hasn't read the paywalled article) I see a few things playing out that are not new, mostly it's the same newer, bigger hammer syndrome:
1. Trying to solve the same old inconsequential problems but with new tech
This happens all the time. Eventually you realise that the problem is actually relatively benign. You want the hammer to hit home every time but realise that even if it did there is no real value gained.
2. Trying to solve a problem that's already been solved with existing more established tech
Open calls for ideas in company innovation labs or platforms are full of noise. Some of that noise is always around the same automation problems, usually some kind of extraction or some kind of categorisation in a workflow. Most of the time products exist but people are just unaware. In large companies the capability might already exist in house but there's a "must invent here" bias
3. Trying to solve hard problems without the right domain or technical expertise
There are legitimate problems that appear innovative and novel, maybe ones that have been waiting for this level of capability to come along so they can execute, however, adequate knowledge of the technical domain (fine-tuning, prompt engineering, zero-shot, context) or the problem domain (how to read a cat-scan) limits their ability to make a cobbled together PoC reliable, repeatable, scalable, trusted.
Think of it as some generalist buying a stock rally car to build a new racing team, but doing it all themselves instead of hiring a mechanic to tune it properly, or a driver to give you the mechanic feedback... or the mechanic to tell the driver what they can and can't do at the extremes... or the driver to tell the mechanic to FO and "fix" it. Dialogue.
4. Problem is too niche and hard to communicate effectively
If a project succeeds in an organisation and there's no one around to hear it, did it really succeed?
5. Lack of existing innovation culture, strategy, or clear direction hamstrings any serious attempt
A non-starter. A lot of organisations still can't embed or operationalise their good ideas properly. If they can't do that already, it's unlikely to change here.
6. LLM successfully implemented into existing product but no body notices or cares.
The whole "put a clock in it" from product design or "get it to send email" of software.
7. Hard problems even with the right attitude and team still take time solve effectively with relatively new technology
Ignoring the legitimate institutional roadblocks of assuring privacy, security, safety, ethics etc. It's still early days. Cost of O&M long-term is still kind of uncertain, as are some of the basic parameters like the context window. Increasing the size of it could fundamentally change your approach. Anyone who was building a system before plugins were announced probably needed to re-think a number of things and go back to the drawing board. Sure there are some who will just continue as planned and iterate later, but some will be cautious before locking in.
Lastly, I know personally for me, LLMs have become a large part of enhancing my daily workflow. They have increased both the quantity and quality of my output but the 2 fundamental problems for me are:
1. Remember it's there and to use it. (Can what I'm doing could be done with LLM assistance)
or
2. How to formulate a question or request. (This is a fundamental problem of all "work" and "management" how do you define and communicate effectively?)
It's a nice story you've created here, but, the thing is, we've all used GPT 3.5/4 now and we know it can easily produce code that automates most jobs.
On one hand, they're clearly impressive and useful. They'll get better, but it's not at all clear how much better.
On the other hand, we've had access to 3.5 for 6 months now, and thus far companies love to talk about their AI strategies, and love to roll out "alpha previews", but not much of value seems to have been produced thus far.
You'd imagine that an instantly transformative feature would not sit around in alpha to die. I think that companies are likely discovering that it's not magic, and that there are many problems that it can't solve.
It actually seems like we have passed peak LLM at this point.