Is it a problem for you? If so, and you didn't live there as a child, many people would just suggest that you move. Colonialism isn't history in Hawai'i.

It never really bothered me when I lived on O'ahu. I know the difference between "damn haoles" and "fuckin' haoles", and I know which one I am.