Hacker News new | past | comments | ask | show | jobs | submit | ramesh31's comments login

>the context limits on google are nuts! Being able to pump 2 million tokens in and having it cost $0 is pretty crazy rn.

What's the catch though? I was looking at Gemini recently and it seemed too good to be true.


Your code becomes training data[0]:

> When you use Unpaid Services, including, for example, Google AI Studio and the unpaid quota on Gemini API, Google uses the content you submit to the Services and any generated responses to provide, improve, and develop Google products and services and machine learning technologies, including Google's enterprise features, products, and services, consistent with our Privacy Policy.

[0] https://ai.google.dev/gemini-api/terms


Google inference is a lot cheaper since they have their own hardware so they don't have to pay licensing to NVIDIA, thus their free tier can give you much more than others.

Other than that the catch is like all other free tiers, it is marketing and can be withdrawn at any moment to get you to pay after you are used to their product.


But if the scaling law holds true, more dollars should at some point translate into AGI, which is priceless. We haven't reached the limits yet of that hypothesis.

> which is priceless

This also isn't true. It'll clearly have a price to run. Even if it's very intelligent, if the price to run it is too high it'll just be a 24/7 intelligent person that few can afford to talk to. No?


Computers will be the size of data centres, they'll be so expensive we'll queue up jobs to run on them days in advance, each taking our turn... history echoes into the future...

Yea, and those statements were true. For a time. If you want to say "AGI will be priceless some unknown time into the future" then i'd be on board lol. But to imply it'll be immediately priceless? As in no cost spent today wouldn't be immediately rewarded once AGI exists? Nonsense.

Maybe if it was _extremely_ intelligent and it's ROI would be all the drugs it would instantly discover or w/e. But lets not imply that General Intelligence requires infinitely knowing.

So at best we're talking about an AI that is likely close to human level intelligence. Which is cool, because we have 7+ billion of those things.

This isn't an argument against it. Just to say that AGI isn't "priceless" in the implementation we'd likely see out of the gate.


a) There is evidence e.g. private data deals that we are starting to hit the limitations of what data is available.

b) There is no evidence that LLMs are the roadmap to AGI.

c) Continued investment hinges on their being a large enough cohort of startups that can leverage LLMs to generate outsized returns. There is no evidence yet this is the case.


> c) Continued investment hinges on their being a large enough cohort of startups that can leverage LLMs to generate outsized returns. There is no evidence yet this is the case.

Why does it have to be startups? And why does it have to be LLMs?

Btw, we might be running out of text data. But there's lots and lots more data you can have (and generate), if you are willing to consider other modalities.

You can also get a bit further with text data by using it for multiple epochs, like we used to do in the past. (But that only really gives you at best an order of magnitude. I read some paper that the returns diminish drastically after four epochs.)


Private data is 90% garbage too

"There is no evidence that LLMs are the roadmap to AGI." - There's plenty of evidence. What do you think the last few years have been all about? Hell, GPT-4 would already have qualified as AGI about a decade ago.

>What do you think the last few years have been all about?

Next token language-based predictors with no more intelligence than brute force GIGO which parrot existing human intelligence captured as text/audio and fed in the form of input data.

4o agrees:

"What you are describing is a language model or next-token predictor that operates solely as a computational system without inherent intelligence or understanding. The phrase captures the essence of generative AI models, like GPT, which rely on statistical and probabilistic methods to predict the next piece of text based on patterns in the data they’ve been trained on"


Everything you said is parroting data you’ve trained on, two thirds of it is actual copy paste

He probably didn't need petabytes of reddit posts and millions of gpu-hours to parrot that though.

I still don't buy the "we do the same as LLMs" discourse. Of course one could hypothesize the human brain language center may have some similarities to LLMs, but the differences in resource usage and how those resources are used to train humans and LLMs are remarkable and may indicate otherwise.


Not text, he had petabytes of video, audio, and other sensory inputs. Heck, a baby sees petabytes of video before first word is spoken

And he probably cant quote Shakespeare as well ;)


>Not text, he had petabytes of video, audio, and other sensory inputs. Heck, a baby sees petabytes of video before first word is spoken

A 2-3 year old baby could speak in a rural village in 1800, having just seen its cradle (for the first month/s), and its parents' hut for some more months, and maybe parts of the village afterwards.

Hardly "petabytes of training video" to write home about.


you are funny. Clearly your expertise with babies comes from reading books about history or science, rather than ever having interacted with one…

What resolution of screen do you think you would need to not distinguish from reality? For me personally i very conservatively estimate it to be on above OOM of 10 4k screens by 10, meaning 100k screens. If a typical 2h 4k is ~50gb uncompressed, that gives us about half a petabyte per 24h (even with eyes closed). Just raw unlabeled vision data.

Probably a baby has a significantly lower resolution, but then again what is the resolution from the skin and other organs?

So yes, petabytes of data within the first days of existence - well, likely before even being born since baby can hear inside the uterus, for example.

And very high signal data, as you’ve stated yourself (nothing to write home about) mainly seeing mom and dad, as well as from a feedback loop POV - a baby never tells you it is hungry subtly.


> he had petabytes of video, audio, and other sensory inputs

He didn't parrot a video or sensory inputs though.


No, they don’t - they don’t have the hardware, yet. But they do parrot sensory output to eg muscles that induce the expected video sensory inputs in response, in a way that mimics the video input of “other people doing things”.

And yet with multiple OoM more data he still didn't cost millions of dollars to be trained nor multiple lifetimes in gpu-hours. He probably didn't even register all the petabytes passing through all his "sensors", those are some characteristics that we are not even near understanding and much less replicating.

Whatever is happening in the brain is more complex as the perf/cost ratio is stupidly better for humans for a lot of tasks in both training and inference*.

*when considering all modalities, o3 can't even do the ARC AGI in vision mode but rather just json representations. So much for omni.


>Everything you said is parroting data you’ve trained on

"Just like" an LLM, yeah sure...

Like how the brain was "just like" a hydraulic system (early industrial era), like a clockwork with gears and differentiation (mechanical engineering), "just like" an electric circuit (Edison's time), "just like" a computer CPU (21st century), and so on...

You're just assuming what you should prove


What do you think "AGI" is supposed to be?

o1 points out this is mostly about “if submarines swim”.

https://chatgpt.com/share/6768c920-4454-8000-bf73-0f86e92996...


This comment isn't false but it's very naive.

You have described something but you haven't explained why the description of the thing defines its capability. This is a tautology, or possibly a begging of the question, which takes as true the premise of something (that token based language predictors cannot be intelligent) and then uses that premise to prove an unproven point (that language models cannot achieve intelligence).

You did nothing at all to demonstrate why you cannot produce an intelligent system from a next token language based predictor.

What GPT says about this is completely irrelevant.


>You did nothing at all to demonstrate why you cannot produce an intelligent system from a next token language based predictor

Sorry, but the burden of proof is on your side...

The intelligence is in the corpus the LLM was fed with. Using statistics to pick from it and re-arrange it gives new intelligent results because the information was already produced by intelligent beings.

If somebody gives you an excerpt of a book, it doesn't mean they have the intelligence of the author - even if you have taught them a mechanical statistical method to give back a section matching a query you make.

Kids learn to speak and understand language at 3-4 years old (among tons of other concepts), and can reason by themselves in a few years with less than 1 billionth the input...

>What GPT says about this is completely irrelevant.

On the contrary, it's using its very real intelligence, about to reach singularity any time now, and this is its verdict!

Why would you say it's irrelevant? That would be as if it merely statistically parroted combinations of its training data unconnected to any reasoning (except of that the human creators of the data used to create them) or objective reality...


Let's pretend it is 1940

Person 1: rockets could be a method of putting things into Earth orbit

Person 2: rockets cannot get things into orbit because they use a chemical reaction which causes an equal and opposite force reaction to produce thrust'

Does person 1 have the burden of proof that rockets can be used to put things in orbit? Sure, but that doesn't make the reasoning used by person 2 valid to explain why person 1 is wrong.

BTW thanks for adding an entire chapter to your comment in edit so it looks like I am ignoring most of it. What I replied to was one sentence that said 'the burden of proof is on you'. Though it really doesn't make much difference because you are doing the same thing but more verbose this time.

None of the things you mentioned preclude intelligence. You are telling us again how it operates but not why that operation is restrictive in producing an intelligent output. There is no law that saws that intelligence requires anything but a large amount of data and computation. If you can show why these things are not sufficient, I am eager to read about it. A logical explanation would be great, step by step please, without making any grand unproven assumptions.

In response to the person below... again, whether or not person 1 is right or wrong does not make person 2's argument valid.


It's not like we discovered hot air ballons, and some people think we'll get to Moon and Mars with them...

> Does person 1 have the burden of proof that rockets can be used to put things in orbit? Sure, but that doesn't make the reasoning used by person 2 valid to explain why person 1 is wrong.

The reasoning by person 2 doesn't matter as much if 1 is making an ubsubstantiated claim to begin with.

>There is no law that saws that intelligence requires anything but a large amount of data and computation. If you can show why these things are not sufficient, I am eager to read about it.

Errors with very simple stuff while getting higher order stuff correct shows that this is not actual intelligence matching the level of performance exhibited, i.e. no understanding.

No person who can solve higher level math (like an LLM answering college or math olympiad questions) is confused by the kind of simple math blind spots that confuse LLMs.

A person understanding higher level math, would never (and even less so, consistently) fail a problem like:

"Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks double the number of kiwis he did on Friday, but five of them were a bit smaller than average. How many kiwis does Oliver have?"

https://arxiv.org/pdf/2410.05229

(of course with these problems exposed, they'll probably "learn" to overfit it)


> The reasoning by person 2 doesn't matter as much if 1 is making an ubsubstantiated claim to begin with.

But it doesn't make person 2's argument valid.

Everyone here is looking at the argument by person 1 and saying 'I don't agree with that, so person 2 is right!'.

That isn't how it works... person 2 has to either shut up and let person 1 be wrong in a way that is wrong, but not for the reasons they think, or they need to examine their assumptions and come up with a different reason.

No one is helped by turning critical thinking into team sports where the only thing that matters is that your side wins.


The delta-V for orbit is a precisely defined point. How you get there is not.

What is the defined point for reaching AGI?


I can check but I am pretty sure that using a different argument to try and prove something is wrong will not make another person's invalid argument correct.

Person 3: Since we can leave earths orbit, we can reach faster than light speed, look at this graph over our progress making faster rockets we will for sure reach there in a few years!

So there is a theoretical framework which can be tested against to achieve AGI and according to that framework it is either not possible or extremely unlikely because of physical laws?

Can you share that? It sounds groundbreaking!


The people who claim we'll have sentient AI soon are the ones making the extraordinary claims. Let them furnish the extraordinary evidence.

So, I think people in this thread, including me, have been talking past each other a bit. I do not claim that sentient AI will emerge. I am arguing that the person who is saying that it can't happen for a specific reason is not considering that the reason they are stating implicitly is that nothing can be greater than the sum of its parts.

Describing how an LLM operates and how it was trained does not preclude the LLM from ever being intelligent, and it almost certainly will not become intelligent, but you cannot say that it didn't for the reasons the person I am arguing with is saying, which is that intelligence can not come from something that works statistically on a large corpus of data written by people.

A thing can be more than the sum of its parts. You can take the English alphabet, which is 26 letters, and arrange those letters along with some punctuation to make an original novel. If you don't agree that means that you can get something greater than what defines it components, then you would have to agree that there are no original novels because they are composed of letters which were already defined.

So in that way, the model is not unable to think because it is composed of thoughts already written. That is not the limiting factor.


> If somebody gives you an excerpt of a book, it doesn't mean they have the intelligence of the author

A closely related rant of my own: The fictional character we humans infer from text is not the author-machine generating that text, not even if they happen to share the same name. Assuming that the author-machine is already conscious and choosing to insert itself is begging the question.


Have you ever heard of a local maxima? You don't get an attack helicopter by breeding stronger and stronger falcons.

For an industry that spun off of a research field that basically revolves around recursive descent in one form or another, there's a pretty silly amount of willful ignorance about the basic principles of how learning and progress happens.

The default assumption should be that this is a local maximum, with evidence required to demonstrate that it's not. But the hype artists want us all to take the inevitability of LLMs for granted—"See the slope? Slopes lead up! All we have to do is climb the slope and we'll get to the moon! If you can't see that you're obviously stupid or have your head in the sand!"


So far we haven't even climbed this slope to the top yet. Why don't we start there and see if it's high enough or not first? If it's not, at the very least we can see what's on the other side, and pick the next slope to climb.

Or we can just stay here and do nothing.


You’re implicitly assuming only a global maximum will lead to useful AI.

There might be many local maxima that cross the useful AI or even AGI threshold.


And we aren't even at a local maximum. There's still plenty of incremental upwards progress to be made.

I never said anything about usefulness, and it's frustrating that every time I criticize AGI hype people move the goalposts and say "but it'll still be useful!"

I use GitHub Copilot every day. We already have useful "AI". That doesn't mean that the whole thing isn't super overhyped.


No, GPT-4 would have been classified as it is today: a (good) generator of natural language. While this is a hard classical NLP task, it's a far cry from intelligence.

GPT-4 is a good generator of natural language in the same sense that Google is a good generator of ip packets.

> GPT-4 would already have qualified as AGI about a decade ago.

Did you just make that up?


A lot of people held that passing the Turing Test would indicate human-level intelligence. GPT-4 passes.

Link to GPT-4 passing the turing test? Tried googling, could not find anything.

Google must be really going downhill. DDG “gpt turing test” provides nothing but relevant links. Here’s a paper: https://arxiv.org/pdf/2405.08007

Probably asked an "AI"

The last four years?

ELIZA 2.0


I agree, these are good points.

Have we really hit the wall?

Do they use GPS based data?

Feels like there’s data all around us.

Sure they’ve hit the wall with obvious conversations and blog articles that humans produced, but data is a by product of our environment. Surely there’s more. Tons more.


We also could just measure the background noise of the universe and produce unlimited data.

But just like GPS data it isn't suited for LLMs given that you know it has no relevance what so ever to language.


Ignoring the confusion about 'GPS' for a moment: there's lots and lots of other data that could be used for training AI systems.

But, you need to go multi-modal for that; and you need to find data that's somewhat useful, not just random fluctuations like the CMB. So eg you could use YouTube videos, or even just point webcams at the real world. That might be able to give your AI a grounding in everyday physics?

There's also lots of program code you can train your AI on. Not so much the code itself, because compared to the world's total text (that we are running out of), the world's total human written code is relatively small.

But you can generate new code and make it useful for training, by also having the AI predict what happens when you (compile and) run the code. A bit like self-playing for improving AlphaGo.


You’re thinking of language in the strictest of sense.

GPS data as it relates to location names, people, cultures, path finding.


What does culture and names and people have to do with the Global Position System?

You are right that we can have lots more data, if you are willing to consider other modalities. But that's not 'GPS'. Unless you are using an idiosyncratic definition of GPS?


Key to understanding the power of agentic workflows is tool usage. You don't have to write logic anymore, you simply give an agent the tools it needs to accomplish a task and ask it to do so. Models like the latest Sonnet have gotten so advanced now that coding abilities are reaching superhuman levels. All the hallucinations and "jitter" of models from 1-2 years ago has gone away. They can be reasoned on now and you can build reliable systems with them.

> you simply give an agent the tools

That isn’t simple. There is a lot of nuance in tool definition.


Depends on what you’re building. A general assistant is going to have a lot of nuance. A well defined agent like a tutor only has so many tools to call upon.

Use Tailwind. It's a massive difference from just asking the LLM to write raw CSS. Tailwind provides a semantic layer that allows them to actually understand it.

Grappling with this hard right now. Anyone who is still of the "these things are stupid and will never replace me" mindset needs to sober up real quick. AGI level agentic systems are coming, and fast. A solid 90% of what we thought of as software engineering for the last 30 years will be completely automated by them in the next couple years. The only solution I see so far is to be the one building them.

As someone who's personally tried ( with lots of effort) to build agentic assistants/systems 3+ times over the course of the last few years I haven't seen any huge improvements in the quality of output. I think you greatly underestimate the plateau these models are running into.

Grok and o1 are great examples of how these plateaus also wont be overcome with more capital and compute.

Agentic systems might become great search/research tools to speed up the time it takes to gather (human created) info from the web, but I don't see them creating anything impressive or novel on their own without a completely different architecture.


>As someone who's personally tried ( with lots of effort) to build agentic assistants/systems 3+ times over the course of the last few years I haven't seen any huge improvements in the quality of output. I think you greatly underestimate the plateau these models are running into.

As someone who's personally tried with great success to build agentic systems over the last 6 months, you need to be aware of how fast these things are improving. The latest Claude Sonnet makes GPT-3.5 look like a research toy. Things are trivial now in the code gen space that were impossible just earlier this year. Anyone not paying attention is missing the boat.


>As someone who's personally tried with great success to build agentic systems over the last 6 months.

Like what? You're the only person ive seen claim they've built agentic systems with great success. I dont regard improved chat-bot outputs as success, im talking about agentic systems that can roll their own auth from scratch, or gather data from the web independently and build even a mediocore prediction model with that data. Or code anything halfway decently in something other than Python.


> There was the rather strange Wake Up, Ron Burgundy: The Lost Movie, released shortly after the feature film came out in 2004. Essentially, director Adam McKay had shot so much material while making Anchorman: The Legend Of Ron Burgundy, and abandoned so many story ideas (including a whole subplot about a fictional terrorist organisation) that it was cut into a separate 90-minute release. (It wasn’t great, but an interesting curio nonetheless.)

I miss when studios and directors would actually edit movies properly. If Anchorman were released today on Netflix, it would have been a 2 hour 15 minute slog, with 10 minutes of laughs here and there, rather than the sharp and hilarious 90 minute comedy that it was.


The Business Plot finally worked

Workflow management. The disparate flow of actions is a nightmare.


ChatGPT is just terrible at code in general. Claude feels like a generational leap above it.


One of the very best episodes of In Our Time: https://www.bbc.co.uk/sounds/play/b0bkpjns


I love In Our Time so much. I often wonder who, if anyone, could take over from Melvin when he's gone.


Brilliant, thank you so much


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: