Hacker News new | past | comments | ask | show | jobs | submit login

>> so far o1-mini has bodied every task people are saying LLMs can’t do in this thread

> give me a query and i’ll ask it

Here's a query similar to one that I gave to Google Gemini (version unknown), which failed miserably:

---query---

Steeleye Span's version of the old broadsheet ballad "The Victory" begins the final verse with these lines:

Here's success unto the Victory / and crew of noble fame

and glory to the captain / bold Nelson was his name

What does the singer mean by these lines?

---end query---

Italicization is for the benefit of HN; I left that out of my prompt.




Hmm... Gemini (1.5 Flash) just aced that exact question for me:

These lines celebrate the victory of the British ship HMS Victory, led by the famous Admiral Lord Nelson, in the Battle of Trafalgar in 1805.

"Here's success unto the Victory": This line directly praises the ship itself, acknowledging its role in the successful battle. "and crew of noble fame": This recognizes the bravery and skill of the sailors who served aboard the Victory. "and glory to the captain": This line specifically honors Admiral Nelson, the captain of the Victory, for his leadership and strategic brilliance in the battle. "bold Nelson was his name": This emphasizes Nelson's courage and daring, which were legendary. The lines express admiration for the ship, its crew, and most importantly, Admiral Nelson, who became a national hero in Britain for his victory at Trafalgar.


That's not acing the question. It's completely incorrect.

What do you think the singer in "Friends in Low Places" meant in the toast he gave after crashing his ex-girlfriend's wedding?

And I saw the surprise and the fear in his eyes

when I took his glass of champagne

and I toasted you, said "Honey, we may be through

but you'll never hear me complain"


That requires knowing the song, beyond the words provided. Would you flunk an eighth grader for getting it wrong?


Well, the rest of the song helps, in that it specifies that (1) the toast upset the wedding, and (2) the singer responded to that by insulting "you", which is presumably one or more of the bride, the groom, and the guests.

But I think specifying that the singer has crashed his ex-girlfriend's wedding is already enough that you deserve to fail if your answer is "he says he's not upset, so what he means is that he's not upset". It's not any kind of leap to guess that the bride's ex-boyfriend's toast might cause a scene at a wedding - that's why the bride's ex-boyfriends are never invited.

(The question has already provided every word of the toast that appears in the song.)

See also the sidethread comment by mikeruiz, noting that o1-pro reproduces the rest of the lyrics to The Victory, but gets the question wrong anyway.


I was referring to the original query, of course, as any entity capable of reasoning could have figured out.


Hmm. Is there anything in my comment above that might address that point of view?


Nah, intermittent failures are apparently enough to provide evidence that an entire class of entities is incapable of reason. So I think we've figured this one out...


Sounds like you just proved ted_dunning isn't sentient.


Well, I proved that he's happy to express an opinion on whether an answer to a question is correct regardless of whether he knows anything about the question. I wouldn't trust advice from him or expect his work output to stand up to scrutiny.

Sentience isn't really a related concept.


i'd prefer an easily verifiable question rather than one where we can always go "no that's not what they really meant" but someone else with o1-mini quota can respond


“They’re toasting Admiral Nelson’s ship (HMS Victory) and its valiant crew, hailing the ship’s successes and Nelson’s heroism. In other words, the singer is offering tribute—“success unto the Victory”—to the vessel and its famed sailors, and “glory to the captain” who led them, namely the celebrated Admiral Horatio Nelson.”

…but to your point, no idea if the artist intended some more obscure reference.

o1-pro was also able to produce a relatively complete version of original source, though, amusingly, referred to it as a ‘broadside’ rather than ‘broadsheet’. Appropriate given the context!


> no idea if the artist intended some more obscure reference.

No, but the answer is obviously wrong. Very, very wrong. The lines are not praise. The song's view of the Victory, and of the battle of Trafalgar, is overwhelmingly negative.

> o1-pro was also able to produce a relatively complete version of original source

...as should be clear, if you read the original source. (Or, heck, just the two lines following the ones in the question.) Being able to reproduce the source makes the error look that much worse.

> though, amusingly, referred to it as a ‘broadside’ rather than ‘broadsheet’.

Both terms are used. https://en.wikipedia.org/wiki/Broadside_ballad :

>> A broadside (also known as a broadsheet) is a single sheet of inexpensive paper printed on one side, often with a ballad, rhyme, news and sometimes with woodcut illustrations.


It's not a difficult or tricky question.


i think it's a bit tricky, the surface meaning is extremely praiseworthy and some portion of readers might interpret as someone who has praise for Admiral Nelson but hates the press gangs.

of course, it is a sardonic, implicit critique of Admiral Nelson/the victory, etc. but i do think it is a bit subtle.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: