Interesting. I find the same thing with music itself. I often hate the albums I end up liking the most the first few times I listen to them. Possibly it's because those artists I make the effort to listen to when I don't like their new albums at first are evolving their sound rather than trying to rehash old successes.
Anyway, my first impression of new music often doesn't match what my later opinion will be after listening more.
This is the same for me. I've also learned how to recognize an album that I WILL like after more listening, which feels really weird, but it works.
Also, very catchy albums that immediately grab me rarely prove to have long-term listenability for me. An example would be the Postal Service album, I remember thinking that was really catchy and great, but after a few listens I hated it.
Anyway, my first impression of new music often doesn't match what my later opinion will be after listening more.