They matter in what problems each field focuses on. And the idea that the natural numbers are all the same, but infinities have different character can be seen as arbitrary as well (cf. set theory).