Because you never know how often the AP will broadcast it's SSID so the passive listener doesn't know how long to listen on any particular channel. In the case where the AP doesn't broadcast the SSID it's even worse, the passive listener relies on other clients actively scanning and the AP responding with its SSID.