It definitely would. "Okay, that's the fourth and final interview; you're hired, salary is within our range, now give us your urine so we can see if you're a heroin addict" is typically not how I like to start major employee/employer trust relationships.

It's certainly not personal. I just think it's bad policy for all but the most safety-critical occupations, and choose to avoid employers who insist on collecting my bodily fluids or health history.