Are you sure you are not mixing up mathematics with astrology? The term 'mathematician' (or better - its equivalent in the day's languages) was often used as a synonym for astrologer.

Maybe they also confused it at the time and all mathematicians were associated with astrologers?

Surely this must be a decided thing in history...