A non rating doesn't directly affect their score. But if a driver doesn't get plenty of five star reviews, they'll get dropped from the app because there are always people who give bad reviews because they had a bad day.