There's also a bunch of stuff in drivers/ that would be excluded in a typical x86 build, though it (mostly) isn't explicitly architecture-dependent. Over half of the whole kernel LOC are in the drivers directory, which includes support for all sorts of unusual devices.
I was curious so I did make allmodconfig (I'm on 64-bit x86) and 'made bzImage modules' and counted lines of c/asm source with a corresponding .o file, all the header files and Makefile/Kconfig lines and got:
