Out-of-order just means it can rearrange the decoded uops in a way to keep the execution units at full capacity. So, if an instruction needs the ALU, but it’s busy, and the next one needs the AGU (address generation unit) and doesn’t depend on the results of the ALU one, it can “dispatch” the AGU one while the ALU one waits for the pipeline to move.
Speculative execution refers more towards the decoder/uop generation side of the processor (the “in-order” side). A normal “in-order” processor, upon encountering a conditional jump, would wait until the pipeline is finished to check if it should jump or not. It does it by inserting “bubbles” into the pipeline - essentially doing nothing but waiting.
Speculative execution (or branch prediction) would say, “I think the branch will be taken based on X, Y, Z,” and then keep the pipeline full in the process. If the prediction was right, congratulations! You just saved dozens of clock cycles that otherwise would’ve been wasted. If it was wrong, no worries. The pipeline is then flushed; all the speculated instructions’ results are tossed (before they’re “written back”). Then the processor resumes operation on the correct branch.
Speculative execution doesn’t necessitate an out-of-order architecture, and visa-versa. Just a pipelined one. It’s perfectly possible to have an out-of-order architecture that doesn’t speculate, or a speculative one that is completely “in-order”, but they work hand-in-hand, and it makes sense to have both if you have one.
Is it safe to say speculation is about what to do with instructions following conditionals, and OoO is about what to do with instructions following non-conditionals?
