By that same logic, you might as well just use Fastmail and save yourself the hassle of running your own mailserver altogether - their cheapest plan is $3 a month (or $2.5 a month if you pay for a year at a time).
It's not at all a bad idea. When I had a look at all the pricings it seemed to be price per user. I have a few domains and loads of "users" on my $2.5 server. It's only me using it but I like to have different addresses for different things and it doesn't look like other hosts support this without paying the per user cost.
You can have multiple aliases for the same user. I have about 20-30 email addresses (across multiple domains) from my previous setup all aliased to my main FastMail account.
On top of that, FastMail has a killer feature if you like have different addresses for different things. For example, say your email is name@example.com and somewebsite.com is asking you to provide an email address.
You can just put in somewebsite@name.example.com and FastMail will automatically route that to your mail email as if you had used name+somewebsite@example.com. You don't need to do anything beforehand and can just create these emails on the fly when you encounter a sign-up form.
It simplifies things greatly and if you start getting spam on one, you can just block it.
