Yes, thought a lot, and coded a little. Main problem: sparse bundle adjustment is slow and non-incremental. From what I hear, the same problem is delaying the release of MS Photosynth. Maybe the answer is running SBA on small numbers of user-supplied points rather than SIFT output. Maybe I'm just clueless.