This is not really the same, but may be interesting to some: I subscribe to ChatGPT plus for a month to check out GPT-4. The rate limits were cumbersome though and it can be easy to waste a prompt, so I started to bootstrap:
I would explain my problem to 3.5 and ask it to suggest comprehensive prompts to use with 4 to maximize my limited quota. It worked very well.
In the long years to come the most advance AIs may become so far removed from us that the best intermediaries will be their less advanced brethren.
I use GPT-4 with ChatGPT daily for coding. Here’s what I’ve found works for making the most out the limited prompts OpenAI gives you.
- When you first start, tell ChatGPT about the limitation. Example - “I’m only allowed to send you 25 prompts every 3 hours. As a result, I need you to keep track of the messages I send you. At the end of each response you give me, please tell me how many prompts I have left, ex: this ‘X/25’”
- Combine multiple questions into one prompt, and tell ChatGPT how to handle it. Example “I need help doing X, Y and Z. Please provide all of the details I need for each task, separating your answer for each with ‘—————‘“. Your prompts can be quite long, so don’t hesitate to jam a lot into a single prompt. Just be sure to tell ChatGPT how to handle it, or you’ll end up with shallow answers for each.
- Provide an excruciating amount of detail in your prompts to avoid having to waste other prompts clarifying. For example, let’s say I’m encountering an issue in an app I’m building. I will tell ChaGPT what the issue is, share the relevant parts of my code (striping out any details I don’t want to share), and tell it the expected behavior. This could be dozens of lines of code, multiple files, etc. It can all be one prompt.
I didn't know you could make a request like that. However for my own personal privacy and my particular use case in my more professional I can't use it for certain things. It is also highly limited in the ChatGPT interface: If I give it a small psuedo data set with plausbible date (6 columns, 8 rows) and ask it to do something basic like format it into a table it will start to do that but cut off when it gets to row 6, column 5.
I then remind it that it didn't complete the table, it apologizes, reattemps, and still truncates things. It's output is still far shorter than what I receive in some other prompts so it's not purely a length issue. I'd need to have control to tweak its parameters through the API to get it to respond appropriately.
However, while it may not format the table well, it will still keep the full data set in memory and answer questions on it. For example, I told it the data context. I then asked it why one category (row) saw a decline for a specific year (the columns) and it gave a cogent insightful answers that only my boss and I, the domain experts in my organization, would be able to identify so quickly.
I then asked it to make a projection to a following year, where the data was not given, based on the observed data. I did so. The values were reasonable, but I didn't know why it used them, so I asked it where it got them from, why it chose them:
The answers where incredibly cogent, again on the level that an entry level junior domain expert would give. Here's one nearly verbatim for one of them from my memory. It said "I noticed that the values were consistent for most years but dropped significantly for one year, but the most recent year recovered to an even higher level. So I projected a value that was less than the most recent year but still mostly approximated the average outside of the anomalous year. It gave a bullet point explanation like this for each of the eight rows.
I asked in WHY the drop may have occured that year, and again I had told it the data context so it knew what the data was about, and it have 5 bullet pointed paragraphs that, again, would be very solid answers for a junior practitioner in my area of work.
I asked it was specific formula calculations it used for the projections it mentions. It then apologized for not clarifying earlier (lol, it does that a lot) and then proceeded to tell me that its initial projections were qualitative in nature base on observational criteria. That alone is amazing. In then went further though, without more prompting to say something very much like "However if you would like a more quantitative approach the following formula would be a reasonable approach"
It then went on to describe in great detail the formula I could use to calculate the difference from year to year for each row of data and apply that average to the next projected year's #, along with explaining its reasoning in detail for each step it took.
ChatGPT 3.5 gave very basic answers that I suppose might be useful for a basic user looking for basic possibly trends, as long as they understood that it could be total BS and they needed to vet the answers. GPT-4's analysis was spot on
I can't fully express the extreme utility of this. Giving GPT-4 a pre-aggregated data set to have it give some decent insights automatically could save me hours of work reviewing & finding some of the most obvious trends that would be obvious to me when I see them but would require me to look through 10 to 50 columns of data across 20 to 100 rows of data (keep in mind it would be predigested, cleaned and validated data so work has to be done to get to that point.)
But then GPT-4's preliminary observations would bootstrap my ability to digest the rest of it and have a jumping off point to perform more complex analysis. Then I could give it bullet points and have it summarize my findings in a digestible way for my less data-literate audience, all told saving me hours and getting me out from under a huge backlog of work.
It's a use case that would in no way threaten my job, but make me more productive. And I have enough work that I'm hiring a junior data analyst, and would need to do so even with this increased productivity, and so it would not deprive them of a job either.
It truly would (will!) be a game changer in my day to day work. But I do fully acknowledge that it would, in some other areas of work, reduce the # of employees required to fill the available work. And also that to my fear, future versions could make me less relevant as well, though I think that's further off. Domain expertise is an enormous part of my job.
I see this as an example of the reverse: AI is still stupid enough that it takes humans a degree of skill to craft a request which generates the desired output.
Kolmogorov complexity would like a word! It seems intractable for AI to read minds, there should always be some degree of skill involved in prompt writing.
Agreed. But I think we may reach a point where it is difficult to prompt the most advance AI (which may are may note be an LLM, though maybe contain an LLM mode) in a productive way, especially due to computational demands on resources. And so AI's with lesser, cheaper capabilities may be reasonably competent in "understanding"-- a term I user very loosely-- the problem of dealing with more advanced but resource constrained systems and assist in the best practices in prompting them.
I don't know how true it is vs how much PR it is, but Khan Academy's use of LLMs was interesting in that they apparently craft a prompt from the AI itself. Ie a two step process where the AI generates the steps, and then the AI reasons about the result of the steps to attempt and judge the accuracy of the data. This was just a blurb from the Ted talk[1], but i'd be interested in seeing a slightly more in depth explanation about this strategy.
Made me wonder if you could have an recursive cycle of thinking. Where the "AI" prompts itself, reasons on the output, and does that repeatedly with guards such that it will stop if it doesn't judge anymore advancement.
From what I've seen, the problem is not with the GPT models, but the people themselves. Almost everyday I get multiple unstructured, unclear and often without required context requests from people that I need to iterate back and forth with additional questions to get a clear view what they're trying to achieve. The only thing GPT is "stupid enough" is that it's not prompted to ask questions back to clarify the request.
That’s my point though, us humans get better at working with abysmal directions the more we encounter it. Current AI, on the other hand, requires humans to improve to meet AI cannot still take those inputs literally, warts and all.
Thats a great idea. I'm going to start doing this. For me it also seems GPT-4 just prints out slower. I find I can get most done with 3.5 and its faster to achieve what Im looking for. Then when I'm not satisfied with the 3.5 response I can clean it up and feed over into 4.
I kick off with the 4 as there's no time to waste, and solely utilize 3.5 in API mode for my apps. It's way speedier, and if you're certain the task is doable, it's a no-brainer to employ it. Scripted uses are often like that.
Not exactly sure why you bring this up but tangentially this is actually a really good prompt to use with GPT. Ask it a question but tell it to list the known knowns, known unknowns and unknown unknowns before replying. The unknown unknowns part usually generates some interesting follow up questions.
I would explain my problem to 3.5 and ask it to suggest comprehensive prompts to use with 4 to maximize my limited quota. It worked very well.
In the long years to come the most advance AIs may become so far removed from us that the best intermediaries will be their less advanced brethren.