>even the most advanced LLMs, including GPT4, have an incredibly short working memory, while reliable reasoning requires accurate and robust retrieval and integration of all relevant information.

I would say that the most "reliable reasoning requires accurate and robust retrieval and integration" of more than just "all relevant information".

One of the things that makes it so difficult to come by, naturally with human beings as well as anybody else.