I think you should worry. One of my math professors taught us how factoring to a(x+b(x+c*(...))) is a great optimization in an introductory class, because he had no idea about CPU pipelines for example.
I'm working on a project where the evaluation of a 4th degree polynomial is on the hot path. My micro benchmark (evaluating two polynomials + little bit of addition) shows that using simple Horner's rule is already somewhat faster, and implementing Horner's rule with fused-mul-add is more than twice as fast.
