It might be that it's simply Firefox's JavaScript interpreter which hasn't been optimized for 64-bits. As far as I know other JavaScript interpreters don't suffer from this problem, and with LuaJIT, which should be fairly similar to a JavaScript JIT compiler, at least I have the numbers to back it up: http://luajit.org/performance_x86.html
LuaJIT doesn't have increased memory usage. 32 or 64 bit, it always packs pointers into the low bits of NaN doubles. Addresses fit because current 64 bit chips actually use 48 bit addresses.
(LuaJIT also uses a limited memory arena for allocations but it's not inherent to the design; it could be changed but the current garbage collector can't handle that much data)
