We had 6 people when we did our first retreat, so you may have a point. Let me be clear: we were there Monday to Friday, so it was 5 days.

It was important that we did this during the week so that we didn't cut into personal weekend time, e.g. with families, spouses, and so on.