Daniel Lemire has an observation here: our brains don't use all the connections, all the time (when it does it's called epilepsy); it would be interesting if we built some computers that also worked like that - with only small parts being active at a certain time, but switching quickly from one to another. That would help with the heating problem and it might allow processors a lot more complex than we have now.
if you're trying to achieve parity with human brain calculation the most important thing is how slow neural connections are. they're snail-paced compared with gigabit links and CPU's @ 3 Ghz+. There's quite a lot of memory that is needed but if you're allowed to make a million round-trips across the room before you miss your realtime deadline, it's kind of silly to think that it won't happen. The fact you point out (about not all parts firing at once) could be useful because it might result in these links not being saturated by data anyway.
From my understanding, this is already present in modern architectures and systems, and will keep increasing . Instead of building only general-purpose circuits (e.g. something like a CPU), you can build lots of special-purpose circuits (multipliers, CRC, FFT, AES, hashing, h.264,...).
Daniel Lemire has an observation here: our brains don't use all the connections, all the time (when it does it's called epilepsy); it would be interesting if we built some computers that also worked like that - with only small parts being active at a certain time, but switching quickly from one to another. That would help with the heating problem and it might allow processors a lot more complex than we have now.