I think you need to have a pretty good understanding of the underlying workings of at least a couple of modern models. You should also follow along closely enough to see if major changes happen in the way they are built.

But watching every new paper? Nah, that's mostly only useful if you have a large enough amount of compute to try them out. And most of us don't have that anyway.