As far as I know, no. I'm about 90% sure you have to rebuild the kernel to disable it. That said, I've only read about it, not played with it, and I don't have a box with it enabled at my fingertips.

That said, this is a local exploit, so unless you have untrusted users or otherwise run untrusted native code, your risk is pretty low. Someone would have to exploit something else first to get access to run custom code on the box.

There's always Ksplice, though it's only really available for Oracle (they also distribute free Ubuntu Desktop and Fedora patches to entice you, but I would be willing to bet you're more interested in server).