I am by no mean knowledgeable in those areas, but my understanding is that using specialized hardware/aggressively optimized code in numpy (or similar tools) would be very hard, though. One of the problem is that CUDA or multi-core optimized numpy would have a significant overhead (obviously of a totally different nature), and would be justified only for very large problems. Multi-code numpy was tried a few years ago by E. Jones, and the overhead was almost never worths it IIRC.

The current solution is to use those through libraries which implement linear algebra, etc... , which can be implemented on top of those architectures. There is obviously quite a bit of excitement around those technologies, and this was one of the most talked area at las scipy conference. Video are available: http://blog.enthought.com/?p=184. Some of them are related to those