Really? I'm not sure that narrow of a definition is the commonly used one.

Wikipedia has it as even broader than my use: https://en.m.wikipedia.org/wiki/Person_of_color

I've generally heard it as either "non-white", "non-white, with emphasis on 'brown' skin tones", or "non-white and not covered by a more specific group" in common real world use, depending on the context.