Genuine question: is there a present or planned value proposition for people like me who already have decent search skills? Or are these really for children/elders who (without making any normative claim about whether this is a good thing or not) can't be arsed to perform searches themselves?
Does someone else have good search skills but mingle traditional search engines with LLMs anyways? Why?
I use LLMs every day but wouldn't trust one to perform searches for me yet. I feel like you have to type more for a result that's slower and wordier, and that might stop early when it amasses what it thinks are answers from low effort SEO farms.
I find myself being unable to search for more complex subjects when I don't know the keywords, specialized terminology, or even the title of a work, yet I have a broad understanding of what I'd like to find. Traditional search engines (I'll jump between Kagi, DuckDuckGo, and Google) haven't proved as useful at pointing me in the right direction when I find that I need to spend a few sentences describing what I'm looking for.
LLMs on the other hand (free ChatGPT is the only one I've used for this, not sure which models) give me an opportunity to describe in detail what I'm looking for, and I can provide extra context if the LLM doesn't immediately give me an answer. Given LLM's propensity for hallucinations, I don't take its answers as solid truth, but I'll use the keywords, terms, and phrases in what it gives me to leverage traditional search engines to find a more authoritative source of information.
---
Separately, I'll also use LLMs to search for what I suspect is obscure-enough knowledge that it would prove difficult to wade through more popular sites in traditional search engine results pages.
> I find myself being unable to search for more complex subjects when I don't know the keywords, specialized terminology, or even the title of a work, yet I have a broad understanding of what I'd like to find.
For me this is typically a multi-step process. The results of a first search give me more ideas of terms to search for, and after some iteration I usually find the right terms. It’s a bit of an art to search for content that maybe isn’t your end goal, but will help you search for what you actually seek.
LLMs can be useful for that first step, but I always revert to Google for the final search.
Yeah this is exactly how I use LLMs + Google as well. I would even go further and say that most of the value of Google to me is the ability to find a specific type of source by searching for exact terminology. I think AI search is fatally flawed for this reason. For some things generic factual information is okay ("What's the capital of France?") but for everything else, the information is inextricably bound up with it's context. A spammy SEO blog and a specialist forum might have identical claims, but when received from the latter source it's more valuable, it's just higher signal.
Google used to care about this but no longer does, pagerank sucks and is ruined by SEO, but it still "works" because if you're good you can guess the kind of source you're looking for and what keywords might surface it. LLMs help with that part but you still need to read it yourself, because they don't have theory of mind yet to make good value judgements on source quality and communicate about it.
I also find some use for this. Or I often ask if there's a specific term for a thing that I only know generally, which usually yields better search results, especially for obscure science and technology things. The newer GPTs are also decent at math, but I still use Wolfram Alpha for most of that stuff just because I don't have to double check it for hallucinations.
You might like what we're building in that sense :D (full disclosure, I'm the founder of Beloga). We're building a new way for search with programmable knowledge. You're essentially able to call on search from Google, Perplexity other search engines by specifying them as @ mentions together with your detailed query.
I don't overuse LLMs for now; however when I have a complex problem that would require multiple of searches and dozens of tabs opened and reading through very long docs, asking LLM allows me to iterate order of magnitude faster.
Things that were previously "log a jira and think about it when I have a full uninterrupted day" now can be approached with half an hour spare. This is game changer because "have a full day uninterrupted" almost never happens.
It's like having a very senior coworker who knows a lot of stuff and booking a 30m meeting to brainstorm with them and quickly reject useless paths vs dig more into promising ones, vs. sitting all day researching on your own.
The ideas simply flow much faster with this approach.
I use it to get a high level familiarity with what's likely possible vs what's not, and then confirm with normal search.
I use LLMs also for non-work things like getting high level understanding of taxation, inheritance etc laws in a country I moved in, to get some starting point for further research.
This. Not having to open two dozen tabs and read through so much is a gamechanger, especially for someone who has had trouble focusing with so much open. This is especially true when learning a new technology.
I dunno, I'm not exactly on the AI bandwagon, but search is the one place where I use (and see others using) chatgpt all the time. The fact that Google search has been getting worse for a decade probably helps, but better search -- consistently done, without ads or cruft -- would be worth a few bucks every month for me.
I agree that you can't TRUST them, but half the links regular search turns up are also garbage, so that's not really worse, per se.
Same, but, until recently, I've been using Microsoft's Co-Pilot because for the longest time it did exactly what this new "search" feature added to ChatGPT: it produced a list of source material and links to reference the LLM's output against. It was often instrumental for me and I did begin to use it as a search engine considering how polluted a lot of first-search results have become with spam and empty, generated content.
Oddly, Microsoft recently changed the search version of Copilot to remove all the links to source material. Now it's like talking to an annoying growth-stage-startup middle manager in every way, including the inability to back up their assertions and a propensity to use phrases like "anyway, let's try to keep things moving".
Happy to see this feature set added into ChatGPT – particularly when I'm looking for academic research in/on a subject I'm not familiar with.
I find that my search skills matter less and less because search engines try to be smarter than me. Increasingly I am confronted with largely unrelated results (taking tweaked keywords or synonyms to my query as input apparently) as opposed to no results.
So my conclusion is that the search engines increasingly see the need of search skills as an anti pattern they actively want to get rid of.
On the Google search results page, activate Search tools > All results > Verbatim. You can also create your own search provider bookmark with verbatim search as the default by adding “tbs:li=1” as a query parameter to the Google search URL.
Completely agreed. At a certain point, “skills” became fighting a losing battle with Google incessantly pushing me towards whatever KPIs or ads they’re chasing. It’s a poor use of my effort and time to keep chasing what Google used to be.
I think it’s pretty clear that LLMs can process a document/article/web page faster than any human in order to answer a given question. (And it can be parallelized across multiple pages at once too).
The main hard part of searching isn’t formulating queries to write in the Google search bar, it’s clicking on links, and reading/skimming until you find the specific answer you want.
Getting one sentence direct answers is a much superior UX compared to getting 10 links you have to read through yourself.
Google does offer an AI summary for factual searches and I ignore it as it often hallucinates. Perplexity has the same problem. OpenAI would need to solve that for this to be truly useful
IME Google's summary is not actually hallucinating, the problem is they are forcing it to quote the search results, but they're surfacing bad/irrelevant search results because Google's actual search hasn't worked in years. It's a RAG failure.
For instance I searched for the number to dial to set call forwarding on carrier X the other day, and it gave wrong results because it returned carrier Y.
This is why my most used LLM after code suggestions is Bing. I like that it has lots of references for the things I ask it to double check and read more, but at the same time it can help me dig deeper into a subject rapidly and better formulate the exact question I'm trying to ask and give me a link to the actual data it's getting it's info from.
> Getting one sentence direct answers is a much superior UX compared to getting 10 links you have to read through yourself.
If we assume that people want a 'direct answer', then of course a direct answer is better. But maybe some of us don't want a 'direct answer'? I want to know who's saying what, and in which context, so I can draw my own conclusions.
I use GPT for things that would require multiple Google searches (research). Some examples..
- I count calories... eat out always and at somewhat healthy chains (Cava, Chipolte, etc). Tell GPT (via voice while driving to & or after eating) what ive eaten half the day at those places and then later for dinner. It calculates a calorie count estimation for half the day and then later at dinner the remaining. I have checked to see if GPT is getting the right calories for things off websites and it has.
- Have hiking friends who live an hour or two hours away and we hike once a month an hour or less drive is where we meet up and hike at a new place. GPT suggests such hikes and quickly (use to take many searches on Google to do such). Our drives to these new hikes learned from GPT have always been under an hour.
So far the information with those examples has been accurate. Always enjoy hearing how others use LLMs... what research are you getting done in one or two queries which used to take MANY google searches?
GPT is proving useful for me where something is well documented, but not well explained.
Case in point: Visual Basic for Applications (the Excel macro language). This language has a broad pool of reference material and of Stack Overflow answers. It doesnt have a lot of good explicatory material because the early 2000s Internet material is aging out, being deleted as people retire or lose interest, etc.
(To be frank, Microsoft would like nothing more than to kill this off completely, but VBA exists and is insanely more powerful than the current alternatives, so it lives on.)
I use LLMs as a kind of search that is slightly less structured. There are two broad cases:
1) I know a little bit about something, but I need to be able to look up the knowledge tree for more context: `What are the opposing viewpoints to Adam Smith's thesis on economics?` `Describe the different categories of compilers.`
2) I have a very specific search in mind but it's in a domain that has a lot of specific terminology that doesn't surface easily in a google search unless you use that specific terminology: `Name the different kinds of music chords and explain each one.`
LLMs are great when a search engine would only surface knowledge that's either too general or too specific and the search engine can't tell the semantic difference between the two.
Sometimes when I'm searching I need to be able to search at different levels of understanding to move forward.
It seems good at finding relevant research papers. e.g.
> "Can you provide a list of the ten most important recent publications related to high-temperature helium-cooled pebble-bed reactors and the specific characteristics of their graphite pebble fuel which address past problems in fuel disintegration and dust generation?"
These were more focused and relevant results than a Google Scholar keyword-style search.
However, it did rather poorly when asked for direct links to the documentation for a set of Python libraries. Gave some junk links or just failed entirely in 3/4 of the cases.
I think it's more filling the niche that Google's self immolation in the name of ad revenue started. Besides kagi, there aren't really any solid search engines today (even ddg), and OpenAI has a reach way beyond kagi could dream of outside a billion dollars in marketing.
Even if you are good at writing the queries, Google is so terrible that you end up getting some blogspam etc. in there (or at least I do). A model filtering that out is useful, which I find phind pretty good for. Hopefully this will be even better.
LLMs really make it easy to quickly find documentation for me. Across a huge software project like Mediawiki with so much legacy and caveats, having an LLM parse the docs and give me specific information without me hoping that someone at Stackoverflow did it or if I'm lucky enough to stumble across what I was looking for.
What I really hope this helps solve is covering for the huge lag in knowledge cutoff. A recent example is where it went "oh you're using Go 1.23 which doesn't exist so that's clearly the problem in your Dockerfile, let me fix that".
But I'm not keeping my hopes up, I doubt the model has been explicitly fine-tuned to double check its embedded knowledge of these types of facts, and conversely it probably hasn't even been successfully fine-tuned to only search when it truly doesn't know something (i.e. it will probably search in cases where it could've just answered without the search). At least the behavior I'm seeing now from some 15 minutes of testing indicates this, but time will tell.
Any question that few months ago I would do to stackexchange (or expect and answer from, after a google seqrch) either coding or quantitative, I go to chat gpt now.
I consider myself quite anti LLM hype and I have to admit it has been working amazingly good for me.
The entire tech industry for the last decade (if not more) has been aimed at people who can't be arsed to learn to use computers properly. I would be astonished if this time is somehow different.
I think the skills required will change but more in an adaptation way rather than everything-you-knew-is-now-irrelevant.
I feel like there is a mental architecture to searching where you try and isolate aspects of what you are searching for that are distinct within the broad category of similar but irrelevant things. That kind of mental model I would hope still works well.
For instance consider this query.
"Which clothing outlets on AliExpress are most recommended in forum discussions for providing high quality cloths, favour discussions where there is active engagement between multiple people."
OpenAI search produces a list of candidate stores from this query. Are the results any good? It's going to be quite hard to tell for a while. I know searching for information like this on Google is close to worthless due to SEO pollution.
It's possible that we have at least a brief golden-age of search where the rules have changed sufficiently that attempts to game the system are mitigated. It will be a hard fought battle to see if AI Search can filter out people trying to game AI search.
I think we will need laws to say AI advice should be subject to similar constraints as legal, medical, and financial advice where there is an obligation to act in the interests of the person being advised. I don't want to have AI search delivering the results of the highest bidder.
I know what you mean, but also don't know how it applies here. Not a hater, and not asking rhetorically to dunk on OpenAI. Just haven't found a use for this particular feature.
Which is also exactly something a bad-faith commenter would say, but if I lose either way, I'd rather just ask the question ¯\_(ツ)_/¯
You’re not doing a great job of not rhetorically dunking on OpenAI when you imply that it must be for children, elders, or people who can’t be arsed to search.
The comment was dripping with condescension towards the use of LLM search, and that’s coming from a huge OpenAI skeptic.
Replacing "really for" with "more for e.g" would have been closer to the intended comment. I'll take that L.
Though if I can clarify: "can't be arsed to search" isn't a normative or judgemental claim against anyone, in the same way that "can't remember a phone number/directions" isn't. I'm speaking under the assumption there's a point between now and heat death where massaging search engine queries may literally not be as useful a 'skill' anymore. So there's less utility in young/old people taking the time to learn it.
But I can see how it sounds when I try to squeeze that into a shorter message.
it seems absolutely wonderful for searches that require multiple steps. for example “i want chinese food near me that will be open tomorrow, takes reservations, and has lo mein and will work for my group of two. i want as close as possible if i can but also let me know what reviews say”
my search skills are good but either way that requires 3+ searches and visiting the menu of each restaurant and checking their hours and reservation. remains to be seen if chatgpt search is consistently good at this though
For searches that remain inconclusive, I sometimes double-check with LLMs to see if I have missed anything. It rarely gives relevant new insights, but it’s good to get the confirmation I guess.
I have used Perplexity (and AI search company) a lot and - well I don't think you understand. This is not about it being too difficult to find the information. Its that a search in Google will give you a list of places to go that are relevant to your query. AI search will give you the information you want.
This becomes even better if the information you want is in multiple different places. The canonical question for that used to be "what was the phase of the moon when John Lennon was shot?". There didn't used to be an answer to this in Google - but the AI search was able to break it down, find the date John Lennon was shot (easily available on Google), find the moon phase on that day (again, easily available on Google) and put them together to produce the new answer.
For a more tech relevant example, "what is the smallest AWS EC2 I can run a Tomcat server in?
You 100% can get this information yourself. It just much more time than having an AI do it.
I think this is just the first step for a full-featured agent that not only does searches for you, but also executes whatever was your goal (e.g. a restaurant reservation, etc)
To solve that problem you have to solve all the issues that make me not trust the results. As search, it's fine, since I am perusing and evaluating them. But as an agent, hallucinations and inaccurate answers have to disappear (or very close to disappear).
Does someone else have good search skills but mingle traditional search engines with LLMs anyways? Why?
I use LLMs every day but wouldn't trust one to perform searches for me yet. I feel like you have to type more for a result that's slower and wordier, and that might stop early when it amasses what it thinks are answers from low effort SEO farms.