I don't necessarily do a huge number of breakouts when I'm at a conference anyway. It's mostly meetings, social, and serendipitous interactions.
With virtual conferences, I definitely have a limit both because of distractions and just fatigue. It's one reason I've pretty much refused to pay for any of the conferences that were still trying to charge relatively big bucks last year. I just wouldn't have gotten the same value out of them.
When I go to the in person conferences, it is easy to say I'm out of the office those days, and focus on the conference.
With the online ones, I always end up getting pulled into stuff, but it's ok because I can just listen in the background.