A right of way means that they have priority. In the UK pedestrians only have a right of way in specific cases.
But drivers have a duty of care, and if a pedestrian is crossing a road then obviously they must do what is safe, i.e. slow down and let the pedestrian cross if (s)he is in the way.
At least some US states don't have a legal concept of "right of way", specifically to avoid the tacit implication that the person who has right of way is under no responsibility to avoid a collision if someone in their path doesn't.
e.g., where I live, the traffic laws are defined in terms of who has the responsibility to yield, and undergirded by a catch-all requirement that everybody who is able to act to avoid a collision must do so.
It's perfectly fine to give pedestrians the right of way, because assuming they also have some sense of self preservation isn't really so far fetched.
For instance in Germany they drill into you in driver's ed that when you hit someone with your car, it's pretty much always your fault. I'd say we don't have a jaywalking epidemic.
The reason some US states choose to specify who should yield instead of who gets to go first is because framing things in those terms hopefully alters the ways that drivers think about things in a way that will improve safety overall.
TBH, I suspect that it's an empty gesture that is being used as an alternative to implementing more rigorous driver education and licensure requirements; the US tends to view driving as more of a right than a privilege.
It's reasonable to give priority to pedestrians once they are on the road because it makes sense to allow them to leave the road as quickly as possible instead of being stuck there because cars don't let them, which is both very dangerous and bad for traffic.
It's equally reasonable for pedestrians NOT to have priority to start crossing the road unless there is a clear, marked pedestrian crossing.
Sure, you may get a ticket for not using a pedestrian crossing if there is one nearby, but if you get run over it's still probably the driver's fault. You can have both.
There is a difference between having priority and having the right to be safe.
Priority means that cars must give way, which is the case at pedestrian crossings: Cars must give way BEFORE a pedestrian has started to cross in order to let him cross.
But in any case once a pedestrian is on the road cars have a duty of care not to hit him.
But drivers have a duty of care, and if a pedestrian is crossing a road then obviously they must do what is safe, i.e. slow down and let the pedestrian cross if (s)he is in the way.