With all the options there seems like an opportunity for a single point API that can take a series of prompts, a budget and a quality hint to distribute batches for most bang for buck.
Maybe a small triage AI to decide how effectively models handle certain prompts to preserve spending for the difficult tasks.
It is not so much a drop in quality as there are tasks that every model above a certain threshold will perform equally.
Most can do 2+2 = 4.
One test prompt I use on LLMs is asking it to produce a JavaScript function that takes an ImageData object and returns a new ImageData object with an all direction Sobel edge detection. Quite a lot of even quite small models can generate functions like this.
In general, I don't even think this is a question that needs to be answered. A lot of API providers have different quality/price tiers. The fact that people are using the different tiers should be sufficient to show that at least some people are finding cases where cheaper models are good enough.
I've encountered plenty of tasks where lower quality models work quite well. I prefer using Claude 3 Opus, DBRX, or Llama-3, but that level of quality isn't always needed. Here are a few examples.
Top story picker. Given a bunch of news stories, pick which one should be the lead story.
Data viz color picker. Given a list of categories for a chart, return a color for each one.
Windows Start menu. Given a list of installed programs and a query, select the five most likely programs that the user wants.
Every single use case of LLMs inherently sacrifices quality, whether the developers are willing to admit it or not. I agree with you though that there aren't many use cases where end users would knowingly accept the trade off.
Maybe a small triage AI to decide how effectively models handle certain prompts to preserve spending for the difficult tasks.
Does anything like this exist yet?