You need to get something that explicitly says what you want; if you want the freedom to work on one particular open source project then get them to give you that clause; but also thinkw hether you want it more general; It's very difficult because my understanding is you can't even do drive-by fixes in random projects without an OK.
I've gotten OKs before, but they were verbal and not written, basically my manager at the time stating they will "close an eye" and let me do it. The problem was the other eye was wide open and they did enforce their "all your code are belong to us" stance against me once (you could say I was in a similar position to OP just on a waaay lower magnitude, reason why I'm so invested in this right now).
For another project I've asked them to write me an email from their legal department stating I own copyright for that code and they won't try to claim it. They've done that after a few months passed... It's a small victory I guess.
But I want a proper solution we could all agree on (both as employees and as employers as both are relevant to me right now). Currently I'm gathering all the feedback in this thread and I would love to receive copy pastes from existing employment contracts to formulate a strategy.