I was on a polyphasic schedule for 3 months while in college last year. I noticed I had slightly decreased strength and energy levels while lifting and playing various sports. Overall my body felt slightly sluggish, like it was running at 90% speed. One thing to consider is the fact you are staying up for significantly longer. My diet changed quite a bit because of this - I was eating more often, but smaller meals. I assume this alone could have been effecting my metabolism / energy levels over the long term.

Overall though, it was the negatives on the social side of things, rather than the physical, that caused me to stop.