What I like to think about with GPT is to what extent this just shines light on how we as humans compose text. For instance, that kind of long-term association/reuse process you describe sounds like a stylistic technique a human would use. Sure, GPT doesn't have full understanding of why it uses the technique, but watching it "almost do it right" is a great way to learn how style works in isolation, imo.