This would work for very small companies...but I'm not sure how one can avoid product cycles forever, even without scrum masters and the like. More to the point, how can you make a good product without something approximating product cycles?
The point is not exactly product cycles, but some way to track progress. Jane street also tracks progress and for many people it's the direct profit someone made for the firm. For some it is improving engineering culture so that other people can make better profits.
The problem with safety is that no one knows how to track it, or what they even mean by it. Even if you ignore tracking, wouldn't one unsafe AGI by one company in the world nullifies all their effort? Or safe AI would somehow need to take over the world, which is super unsafe in itself.