Interesting articles. For the size estimation, it might be interesting to measure what a perfect estimate would give you. Because in theory that estimate could be done based on builds in a trusted environment and saved into Cargo, especially for large popular crates.
I’m also curious if it might be possible to pipeline the CGU stage so that you feed the data from rustc to LLVM incrementally as you lower MIR into LLVM IR without having the entire CGU upfront. That may help reduce total processing time by hiding brief IO bubbles that take up a lot of time in aggregate (yes it’s CPU bound, but there’s still going to be implicit IO like memory) and also reduce total peak memory usage for hopefully obvious reasons. To gain full benefit though you might need the entire thing to be pipelined all the way through (from generating the MIR to lowering to LLVM IR) and that may be at odds with things like optimized builds which need to do analysis from a more global perspective (which would similarly inhibit the peak memory usage gains).
I’m also curious if it might be possible to pipeline the CGU stage so that you feed the data from rustc to LLVM incrementally as you lower MIR into LLVM IR without having the entire CGU upfront. That may help reduce total processing time by hiding brief IO bubbles that take up a lot of time in aggregate (yes it’s CPU bound, but there’s still going to be implicit IO like memory) and also reduce total peak memory usage for hopefully obvious reasons. To gain full benefit though you might need the entire thing to be pipelined all the way through (from generating the MIR to lowering to LLVM IR) and that may be at odds with things like optimized builds which need to do analysis from a more global perspective (which would similarly inhibit the peak memory usage gains).