It doesn't has as high an effect as saturated fat, no, but it still has a negative effect, and fact of the matter is that we need to consume zero dietary cholesterol, so every time we do, that is a negative action towards the goal of health.

Another thing to note is that often the case is that food that is high in cholesterol is also high in saturated fat, both things that affect ldl cholesterol levels in a negative way.