I'd recommend just using a project like buildroot - it's really easy to begin with, and afterwards every odd device you eventually decide you need to cross compile for (old router with a MIPS CPU, etc.) becomes just as easy.
I have one of the new Pi's with 1GB ram. With a fast 64GB microsd card ($21 on black friday). I've been pretty impressed with fast it compiles with make -j4.