An alternative approach, IMHO, is to log all the things and just be judicious about expunging old stuff -- I believe the metrics community buys into this approach, too, storing high granularity captures for a week or whatever, and then rolling them up into larger aggregates for longer-term storage

I would also at least try a cluster-local log buffering system that forwards INFO and above as received, but buffers DEBUG and below, optionally allowing someone to uncork them if required, getting the "time traveling logging" you were describing. The risk, of course, is the more chains in that transmission flow the more opportunities for something to go sideways and take out all logs which would be :-(