Currently? Yes. This is a challenging problem for someone with decades of experience. I'm not sure you can train an LLM to appropriately do this because I can't even begin to describe how one would generate an adequate cost function. I don't think even RLHF can resolve that aspect because the truth of the matter is that I don't know what's important in that rabbit hole until I spend time working on the problem, replicating, or have sufficient experience. All too common a single line can make or break an algorithm and that line is 3 papers back. All too common there's nuances that radically change results that aren't even in the papers themselves.

I hope you succeed, but personally I don't know how this could be solved. The problem is that I don't actually need better summarization, its that I need more nuance and technical aspects. The problem exists because we're writing to larger audiences as competition increases and the quality of reviewing decreases (we even have a shortage which only exacerbates this problem). I'm not sure AI solves existential problems that are built around reward hacking, in fact everything I've seen suggests they explicitly do the opposite. I mean we literally train them to do that...