I don't understand - were these models post-trained to play factorio? A) If so, how is that possible given that e.g. Claude doesn't have public weights? B) If not, how would the agent know what the API does? Even if it's "guessing" from the English meaning of the API commands (e.g. place_entity_next_to places entity next to something), how would it know what the recipes are? If it's trying and learning we go back to A).
Having read the pdf I don't think these models were post-trained, so how do we explain the questions in B)?
And if indeed there's no post-training and authors expected exploration of recipes to come from the context window.... I think that's way too short for RL-style improvement.
In short, I don't understand how they could've tested those models with post training, and without post training they all did unbelievably well.
If the authors read this: can you give us an idea how many API query and API pairs fit within the context window, on average? Follow up, do you get better results if you abbreviate the API call names, so that more response pairs fit within one context window?
To also jump in here, regarding tools the agents had access to function signatures (i.e tool docstrings, input and output types) and for each tool a small "manual", which described what the tool does, how it affects the game state and a small number of examples where using this tool would be useful (for instance, how to use place_entity_next_to to put an inserter next to an existing chest)
Overall as Jack said, no post-training was done at all but all agents had a complete API description (tools, entities, research) in their context so the results indicate to some level how well can modern agents use a completely OOD API with decent level of documentation
These models were not post-trained - all off-the-shelf.
We can fit about 128 pairs maximum in the context, but this performed the same as 32, which we ultimately decided on (for cost, latency purposes).
Encoding the input/outputs to make them shorter degraded performance. It seems that descriptive names is helpful for pretrained models because they have an intuition on what they do.
Follow up. Do you have an hypothesis why Claude performs much better than the rest at these tasks?
Is it just because Clause is the best at coding and the API is code? (not very interesting). Maybe if the API required the llms to write in poems, the best LLM at poetry would win...
Or is it because whatever makes claude good at coding, also makes it good at mathematical-like tasks. This is more interesting, as it would show some transfer learning. It would also suggest if you're doing training for a specific task, you would also benefit from training adjacent tasks e.g. if you're training for maths you could benefit from training coding. I believe this is actually true for humans.
And would you know how to check whether if any of the above hypothesis is correct?
> Russia was promised by the US that NATO would not expand. by someone named Baker
This is Russian propaganda. Russia claims this was promised verbally. Surely if this was something that Russia wanted why didn't they demand in writing, like they did about many other concessions made to them? The answer is that Putin made that up. It never happened.
As i said, "Stuff shifted, but under no circumstances was russia going to let Ukraine 'join NATO'." - it's just too close to moscow, and there's been issues in that region since the collapse. "We're going to acquire nukes" was sort of the final straw. Let's just hope that Canada saying the same thing doesn't foment the same response from their neighbor, eh?
I sure can, but why do you ask? I just did 25 minutes of research to give a clear reasoning why they want them, and then finding the dates of when Zelenskyy said they "need" or "should get" nuclear weapons. But i'm unwilling to share this research effort without knowing the purpose behind your asking.
To put it better, what year would be an acceptable answer? Is the year that they loudly claimed "we should have never given up our nuclear weapons" not acceptable? the year "we need Nukes or NATO"?
I also remember sabre rattling in 2014 and either 2016 or 2017. NATO already has missiles pointed at Russia, and Ukraine joining NATO would allow missiles to be launched from even closer to Russia.
I don't understand why people don't get the nuance. The US did dirty. I don't know how to solve that, but the answer isn't "war with russia" or "proxy war with russia" the answer is closer to "get the fighting to stop, and tell russia to chill out and report issues to the world, not sit and stew for 8 years over their ethnic brothers, sisters, and children being 'rained down upon' with death and destruction."
> The far-right candidate Oleh Tiahnybok’s last name means pulling one’s side in Ukrainian.
> So his campaign officers have been conveniently running a message of “Tiahnybok is pulling for our side,” but so far managed to get only 1.6 percent of polled voters to declare support for him. A broad-shouldered and towering leader of the right-wing Svoboda Party, he positions himself as a knight on the yellow-and-blue horse – the country’s national colors – on a mission to save Ukraine. His program almost immediately mentions that a section “nationality” should be introduced into Ukrainian passports – a sign of pride to some, yet prejudice to others. Should he be president, Ukrainians will have to obtain visas to travel to Russia and pass a Ukrainian language test to work in civil service. Ukraine would pick up nuclear arms again and take a hard line approach towards Russia. Serving as a lawmaker twice before, Tiahnybok’s ideas have been better received in the more nationalist west. Once allied with President Victor Yushchenko’s Our Ukraine faction, he was expelled for anti-Semitic and xenophobic statements.
The idea that Russia were forced to invade Ukraine because a guy who ran for President twice getting less than 2% of the vote who whose political party had no seats, proposed obtaining nuclear weapons over a decade earlier is too stupid for anyone to honestly believe.
"\nI bet this isn't going to be good enough, which is why i didn't want to do this. I have more, i just want to prove this point."
to my prior post, the sha256sum will match.
I merely asked what sort of proof you were looking for, buddy. Evidently a Ukrainian politician saying the exact words you said no Ukrainian politician said prior to 2014 isn't good enough.
here, about the party in Ukraine he's led for decades:
> The party gained increasing popularity in the late 2000s and early 2010s, winning 10.45% of the vote in the 2012 parliamentary election. Between 2009 and 2014, it was an observer member of the far-right Alliance of European National Movements. It played a role in the 2014 Ukrainian revolution and Euromaidan protests but its support dropped quickly following the 2014 elections. Since then, the party has been polling below the electoral threshold, and it currently has one seat in the Verkhovna Rada.
oh look, they win parliamentary votes, and hold a seat.
also incaseyoumissedit:
> It played a role in the 2014 Ukrainian revolution and Euromaidan protests
I think it's pretty obvious that the fact a single individual who's ceased even trying to run for office because he can't get the votes and actively despises the actual Ukraine leadership said something 15 years ago is not "evidence" for your claim that '"we're going to acquire nukes" was the final straw'. A guy who literally has less impact on Ukrainian nuclear policy than George Galloway on that of the UK.
But yes, congratulations on predicting that nobody would be impressed by such a stupid irrelevance. The non-stupid thing would not to have posted it in the first place...
When a group of Americans were mad that their preferred candidate didn't win and stormed the Capitol, were they heroes or traitors? When a group of Ukrainians did the same, why does your answer flip?
Euromaidan was a protests because Yanukovych didn't act on his promise to integrate Ukraine with the EU. Protesters were upset because that was not what he promised during his campaign.
The storming of the capital in US was under false claims that the elections were stolen.
There is both evidence that the elections were not stolen, and evidence what Yanukovuch said during his election campaign were not his actions. You can't tell the difference between the two?
Plenty of protests ended in the fall of a government and electing a new one. Why would that be the same as the US storming of the Capital?
If you would have said to compare it to the Georgian protests that don't accept the election results, that would be more difficult. But Euromaidan? That's easy.
Trump lost fairly, so storming the capitol is treason. Yanukovich didn't win fairly, so removing him is warranted. The answer flips because the situations are the opposite.
The people who stormed the Capitol didn't think Trump lost fairly. So the real difference is whether they succeeded. Had the Capitol protesters succeeded, they would have made the media feed people like you and I their narrative and make sure we saw them as freedom fighters.
I'm sure you'll disagree with the above, so here's a thought experiment. What would the Euromaidan protesters be called if they had lost?
The protests started when he switched his stance on getting closer to EU. The protesters got him out and had a new election. How can you compare that to not accepting an election result?
Edit: To answer your question: if the Euromaidan protest didn't succeed, it was just a protest like it was now. They would have had another election a bit later, possibly pissing off Russia again (In 2004 Russia poisoned a pro-EU candidate).
and if we're going to point at everything trump says and react with gusto - like canadian politicians saying they need to arm themselves with nukes to prevent invasion, just to name one - then i question the validity of "oh they didn't sign it so it doesn't matter" The secretary of state said to Gorbachev multiple times "not one inch to the east" of the "NATO" base in Germany.
But none of this matters, because Ukraine is 250 miles from Moscow, and that's a lot different than the nearly 600 miles away Poland is. the closest nato point to moscow as it stands right now is 350 miles away.
if you can't see the difference, i'm sorry. A big point of contention for Russia is the "Ethnic Russian" portions of Ukraine, near the border, who would "gladly" be part of russia, but because of an arbitrary border, they cannot.
Baker and Gorbachev talked about the status NATO forces in East Germany until the Soviet withdrawal in 1994. They agreed that only the forces under direct German control would enter East Germany until the last Soviet forces had withdrawn. Their agreement was formalized in Article 5 of the Treaty on the Final Settlement with Respect to Germany.
The relevant article:
ARTICLE 5
Until the completion of the withdrawal of the Soviet armed forces from the territory of the present German Democratic Republic and of Berlin in accordance with Article 4 of the present Treaty, only German territorial defence units which are not integrated into the alliance structures to which German armed forces in the rest of German territory are assigned will be stationed in that territory as armed forces of the united Germany. During that period and subject to the provisions of paragraph 2 of this Article, armed forces of other states will not be stationed in that territory or carry out any other military activity there.
Germany upheld their part of the agreement, the withdrawal went uneventfully, and the agreement was concluded by the end of August 1994.
As you can see, it has nothing to do with whether NATO would accept new members or not. Gorbachev and many other top Soviet/Russian officials have directly refuted this myth, see: https://news.ycombinator.com/item?id=43149963
the link i gave, under _Document 5_, is the transcripts from their conversations.
Where baker said "not one inch east of the Elbe."
I am not sure how much clearer it can be than declassified documents. "That is a lie putin told, it's propaganda!"
no, it's right there. has been available to look at for 9 years.
I really do not give one WHIT what happened afterward, all i care about is refuting falsehoods.
to wit:
A United States Secretary of State named Baker, gave a "cascade of assurances" that NATO would not expand EAST of the Elbe. Repeatedly, the same message was given to the russians.
That is literally all i said. You're arguing something completely different now and like i said, good for you, i don't care. it wasn't what i was talking about or responding to.
There is not a single mention of Elbe in the linked document. However, there are numerous references to the 2+4 agreement, which is the informal name for the treaty I previously cited. That's what they were discussing and that's what they agreed upon. Soviet representatives have confirmed that the discussions were limited solely to the placement of foreign forces in East Germany until the Soviet withdrawal was complete.
As the Soviet foreign minister stated in a 2014 interview, there was simply no reason to discuss NATO beyond that. They did not expect the Warsaw Pact to dissolve. The idea that the world would change so drastically that the Warsaw Pact would dissolve and its members would seek to join NATO was unimaginable at the time.
You are clinging to an erroneous understanding of a few transcript snippets against the words of the direct participants and their actual written agreements.
So you're saying that Baker never gave a "Cascade of Assurances" (not my fucking words) to Gorbachev? Even though there's dozens of documents there, i'm sure you scoured them all to make sure "Elbe" wasn't referenced. You think i pulled that out of my ass?
I want you to answer the simple question, because you have refused.
Is the statement:
Baker verbally assured gorbachev that there would be no eastward expansion
True or false?
That is literally, and when i say literally i mean literally the only thing i was talking to, above.
And it's obvious if you follow the historical context.
In 1990, Germany was still formally under Allied military occupation (since 1945). In the final 2+4 treaty, East Germany and West Germany - the "two" - negotiated with the four Allied powers (UK, France, USA, and USSR) to determine the terms of reunification. Their discussions centered on whether a unified Germany would be fully neutral, partially neutral, or entirely integrated into NATO. In the end, they agreed that reunified Germany could remain in NATO, provided that no foreign troops were stationed in East Germany until Soviet forces had fully withdrawn by the end of 1994.
That's it. There was never any discussion about the broader future of NATO because there was no reason to have one. Germany bordered the Warsaw Pact, and no-one on the Soviet side expected it to dissolve.
there's no way the Russians would lie, except to say that there was a cascade of assurances, but only if putin says that. If gorbachev said it never happened, that's the truth. If baker says he said that, he's lying, because Gorbachev is your baseline of truth. Basically, anyone that supports what you're saying is telling the truth, and the declassified national security documents (why would they have to classify such a thing as a verbal "Cascade of Assurances" about nothing further east?) I guess that's all part of this conspiracy that putin put in place. putin put in, hilarious.
so you just ignore the declassified documents, making this whole thing a waste of my time.
It's not just Gorbachev. The minister of defense Dmitry Yazov also refuted this myth, as did the minister of foreign affairs Eduard Shevardnadze and his successor Andrei Kozyrev, along with many others. You are clinging to your interpretation of a few phrases from meeting notes and other insignificant documents while ignoring the actual signed treaties, their historical context and the recollections of the participants in these events.
Shevardnadze, in an interview with the German magazine Der Spiegel, was as clear as one could be:
SPIEGEL ONLINE: At the end of March 1990, Genscher and the then US Secretary of State James Baker, talked about the fact that there was interest among "central European states" about getting into NATO. You knew nothing of this?
Shevardnadze: This is the first I've heard of it.
SPIEGEL ONLINE: Did you have a conversation with your colleagues in Poland, Czechoslovakia and Hungary about a possible eastward expansion of NATO in the spring of 1990?
Shevardnadze: No, that was never discussed in my presence.
SPIEGEL ONLINE: The German documents give the impression that Moscow counted on the dissolution of both the Warsaw Pact and NATO. Did you really think that would happen?
Shevardnadze: That may have been discussed after I resigned from the ministry of foreign affairs in December 1990. However during my time in office it was not.
SPIEGEL ONLINE: Was the eastward expansion of NATO ever discussed in the inner circles of the Communist Party of the Soviet Union in 1990?
Shevardnadze: The question never came up.
SPIEGEL ONLINE: Did the subject play a role in the ratification process of the Two-Plus-Four agreement (where the signatories included the two Germanys and the four powers that occupied Germany after World War II) that unified Germany?
Shevardnadze: No, there were no difficulties whatsoever with the ratification process.
SPIEGEL ONLINE: Nevertheless, the eastward expansion happened a few years later. Did you feel, at the time, that the German diplomats deceived you?
Shevardnadze: No. When I was the minister of foreign affairs in the Soviet Union, NATO's expansion beyond the German borders never came up for negotiation. To this day I don't see anything terrible in NATO's expansion.
SPIEGEL ONLINE: At the conference in Ottawa on German unity in February 1990, you had five telephone conversations with Gorbachev. Did you discuss a possible NATO enlargement -- beyond the GDR?
Shevardnadze: No. We only had German reunification on the agenda, nothing else.
Thats a report of a conversation, not a legal agreement. If you want a legal agreement, get it in writing. So like I said it's Putin's propaganda that there was some agreement. Thanks for confirming, I'll be sure to use that link in the future.
I say this as proud European: the problem isn't just VC culture, it's European culture. In Europe there's an extreme risk aversion to trying things that have nonzero risk of failing.
I know a tiny bit about this regarding the UK but no other country in Europe:
My mental shortcut has always been "The US never inherited debtor's prison." Historically in the UK at least, getting into a situation where your debts can't be honored was utterly ruinous (this has improved IIUC). In the US, there are strict upper bounds on how much sway creditors can have over you. One could imagine this would result in a chillier credit market when creditors have fewer protections, but ironically, this makes it easier to get credit in the US because creditors don't have another option. Interest on a successful venture is still the quickest path to making one's money grow, so even knowing the debtor could walk away and the worst that would happen is "bankruptcy followed by a judge telling you you get pennies on the dollar of your investment", people still put up the money.
The most obvious example of how failure to pay debt in the US isn't personally ruinous is probably that our current President has filed bankruptcy six times.
(Note: I am speaking broadly and about non-medical debt. Medical debt in the US is ruinous for several significant reasons. But that's generally a non-overlapping concern to most tech-company funding).
Exactly. And people who think that "did the they break ToS?" is the key point here, are perhaps unaware of the rampant dishonesty that apparently became accepted even in government.
Then the consequence of having an X account blocked is not even worth a mention even in passing.
From the perspective of the DOGE detractors, how is this any different than complaining about a serial killer who doesn’t flush the toilet when they’re done?
Exactly. Why is it a story that a private company banned an account for posting content that violates it’s ToS AS WELL AS targets a group of people special to the primary owner of said company? This story is absolutely nothing other than adding synthetic fuel to the fire that already hates Elon.
I just wanted to say: respect for being able to say "sorry, I made a mistake". I hate the fake it till you make it mentality that seems to be the norm now.
I understand the frustration but I don't think it needed to be said (the part about mentality, the thanks is of course cool), because that's still not the norm.
Why do I even bring this message - I want to say that let's not let what we see in the news influence our perception of the real people of the world. Just because fraud and crimes get elevated in the news, does not mean that the common man is a criminal or a fraud. :)
I think the opposite, just like we used to tell people to "go read a book", now we'll tell people to "spend some time with an AI" to get cultured. The more AI time the better for your education.
My kids read plenty and I am indeed always telling them to "go talk to the LLM". It's more work than reading (I suspect), they have to engage more with the topic - ask good questions.
I don’t even let my kids watch YouTube. Imagine what an llm could hallucinate to impressionable minds.
Edit (addition):
How the fuck did we decide that a large language model somehow became artificial intelligence? It’s like claiming a dictionary is intelligent. I just don’t get it.
Having read the pdf I don't think these models were post-trained, so how do we explain the questions in B)?
And if indeed there's no post-training and authors expected exploration of recipes to come from the context window.... I think that's way too short for RL-style improvement.
In short, I don't understand how they could've tested those models with post training, and without post training they all did unbelievably well.
If the authors read this: can you give us an idea how many API query and API pairs fit within the context window, on average? Follow up, do you get better results if you abbreviate the API call names, so that more response pairs fit within one context window?