I used to play quite a lot too. I think the thing that stopped me was the realisation that to get really good, you have to memorise openings and responses to openings. If you don't, you'll be working from a disadvantage from the start (if you play someone who knows the opening responses).

My favourite bit was always the middle-game - fun through pure invention and strategy that can't be laid out in a book for memorisation because of the almost infinite possibilities!

I really the book "My 60 Memorable Games" by Bobby Fischer- it really gave you an insight into a great players' thinking: