x32 is a separate abi, distinct from the normal 64-bit and 32-bit abis.
Most distros don't use it, most don't even offer an x32 install image, but if they enable the kernel option for the x32 abi even without intending to use it, they're vulnerable.
If you're using the x32 abi, either you're using it on purpose, or your distro happens to offer an x32 install image and you downloaded it by accident instead of the normal 32-bit image you probably wanted. In all other cases it should be safe enough to disable the kernel option.
Most distros don't use it, most don't even offer an x32 install image, but if they enable the kernel option for the x32 abi even without intending to use it, they're vulnerable.
If you're using the x32 abi, either you're using it on purpose, or your distro happens to offer an x32 install image and you downloaded it by accident instead of the normal 32-bit image you probably wanted. In all other cases it should be safe enough to disable the kernel option.