Same experience with context size and prompt quality.
Some other things I picked up:
- If you formulate a good prompt with small (but sufficient) context and it still makes mistakes after one attempt to feed the error message back to it, it's probably not going to be able to ever get it, no matter how many iterations you do with it. It will get stuck in a rut forever. Better not to argue with it.
When I catch an agent falling into a “debug loop” I ask it to summarize its understanding of the error. Next, I ask it to shift its context to a higher level in the code and look for patterns across my files which would impact {summary of error).
It isn’t foolproof but it has a much better success rate for me than letting it spin.
Some other things I picked up:
- If you formulate a good prompt with small (but sufficient) context and it still makes mistakes after one attempt to feed the error message back to it, it's probably not going to be able to ever get it, no matter how many iterations you do with it. It will get stuck in a rut forever. Better not to argue with it.
- o1-2024-12-17 is genuinely a big step change.