I honestly think it is that his academic accomplishments in youth combined with the work he has done since is just triggering to people who have their identity wrapped up in being the smart kid. Since he names his companies after himself, knocking him for being "egotistical" is an easy attack to feel better about oneself for not having a PhD from Caltech at 20 with Feynman advising.

I have listened to hours of him on Lex Fridman and he doesn't come off as egotistical or arrogant in the least.