Is it different every time? Otherwise the training could just memorize the answers.

The models never have access to the answers for the private set -- again, at least in principle. Whether that's actually true, I have no idea.

The idea behind Arc-AGI is that you can train all you want on the answers, because knowing the solution to one problem isn't helpful on the others.

In fact, the way the test works is that the model is given several examples of worked solutions for each problem class, and is then required to infer the underlying rule(s) needed to solve a different instance of the same type of problem.

That's why comparing Arc-AGI to chess or other benchmaxxing exercises is completely off base.

(IMO, an even better test for AGI would be "Make up some original Arc-AGI problems.")

This is meant for a lay audience so you should probably just read her research papers.

Also:

> it gives me the (probably flawed) impression that her research isn't the part of her life that's supposed to be important or impressive.

I don't see this at all in the article. There's just some human interest content to make her research more approachable.

All of those analogies were useful in some ways, and LLMs are too.

There's also a progression in your sequence. There were rudimentary mechanical calculating devices, then electrical devices begat electrical computers, and LLMs are a particular program running on a computer. So in a way the analogies are becoming more refined as we develop systems more and more capable of mimicking human capabilities.

If OpenAI has exclusive rights to AI generation for Disney and other IP rights holders, that would create the kind of moat they've been missing so far.

All the more reason it's insane for Disney to be the one giving money!

Disney is buying equity from OpenAI. You frame it as "giving OpenAI money" because you hold a (quite insane) assumption that OpenAI's equity is worth nothing.

> Disney is buying equity from OpenAI.

Can you buy equity from OpenAI without also giving OpenAI a license to use your IP? Even if the equity is worth $1 billion, how much is Disney's IP license worth?

>"how much is Disney's IP license worth"

It unspoken business model is giving an IP license to anyone that can breathe at make a rev share agreement or hefty sum - so, less than you think.

> commoditized

not for disney content. Disney can pick OpenAI as the winner for this by not signing deals and suing anyone else.

These kinds of equity deals tend to include MFN clauses around inference pricing. Ik Anthropic did something similar a couple years ago.

Thats a business agreement not a moat. And you might have rights to generate the characters but they still need to do something. You only have to look at the repeated Disney flops to see they themselves have no ideas.

And if you're the only company with that business agreement. As long as you still have it, it's a...moat.

Well that’s the thing with moats - they don’t just disappear one day.

No business moat is permanent.

These kinds of parternships also throw in free inference with MFN clauses, which make a mutual moat.

A moat doesn't have to be a feature, and equity stakes have been fairly successful moats (eg. Much of AWS's ML services being powered by Anthropic models due to their equity stake in Anthropic).

A moat is a permanent feature protecting a castle against attack. That’s the metaphor. If it’s not their own device intrinsically protecting them then it’s not a moat in my book.

That is not how we use the term "moat" in this context, because competitors eventually converge on offerings within 1-2 years.

I don't need some stuck up HNer telling me about stuff I deal with in my day-to-day job.

Edit: can't reply

> a business deal that can be transferred to a new partner the second it expires is much more temporary

Generally, these kinds of equity deals include an MFN clause.

> That is not how we use the term "moat" in this context, because competitors eventually converge on offerings within 1-2 years.

Then I guess we need a new term because that's not how I interpret the term moat either. To me, ChatGPT chat history is a moat. It allows them to differentiate their product and competitors cannot copy it. If someone switches to a new AI service they will have to build their chat history from scratch.

By comparison a business deal that can be transferred to a new partner the second it expires is much more temporary.

> To me, ChatGPT chat history is a moat.

Every service has a chat history. You are talking about stickiness, which is (roughly) the same for every product.

ChatGPT wins a bit with stickiness because their AI personalizes itself to you over time, in a way that others don't quite do.

A moat is something unique. It can't really be a moat if all services offer it.

But Sam Altman has already said that they need to be able to ignore copyright laws because the Chinese are going to ignore them too. How is access to Disney IP a moat if everyone involved (except Disney) gives no shits about copyright?

Looks like he changed his mind.

Step 3 is always useful (if not necessary) once you reach a certain scale.

Regions seem like a much cleaner and simpler solution to this problem.

Nothing but the benefit is limited if you can’t pass the arena to functions doing lots of allocating.

PHP is: see your changes by refreshing your browser. At least that was it’s initial appeal.

Not the ability to mix SQL injection vulnerabilities into the middle of your HTML?

Regardless, you’re thinking of Perl/CGI. PHP did attract the Perl crowd away from Perl, but it wasn’t for that reason. That was already the norm.

That is at odds with predicting based on recent rates of progress.

The one about LLMs and mental health is not a prediction but a current news report, the way you phrased it.

Also, the boring consistent progress case for AI plays out in the end of humans as viable economic agents requiring a complete reordering of our economic and political systems in the near future. So the “boring but right” prediction today is completely terrifying.

“Boring” predictions usually state that things will continue to work the way they do right now. Which is trivially correct, except in cases where it catastrophically isn’t.

So the correctness of boring predictions is unsurprising, but also quite useless, because predicting the future is precisely about predicting those events which don’t follow that pattern.