strace is the Daddy of debugging especially when you can attach it to an already existing program using it's PID. I have solved so many permissions errors problems with strace where the application itself just failed without logging anything.
Similar experience here, except that I've solved similar issues (and far more complicated issues) with DTrace instead of strace. The range of information that DTrace can retrieve about a running application surpasses anything that strace can retrieve. DTrace can retrieve information on a systemic scale, whereas each strace instance operates on a single process. And the overhead is significantly lower with DTrace.
For many of the systems applications I design for Illumos, I've used DTrace probes as a way of logging very frequent events, on demand (to avoid frequent IO). All of the events that _must_ be logged for the application to function properly, are logged and fsync'd.
I think that in most systems dynamic tracing will eventually replace a significant portion of the logging functionality that people code into their applications.
Either way, if you think strace is sweet, give DTrace a spin.
Some helpful links:
[0] A video demostration of DTrace, by the creator of DTrace.
If the application fails, wouldn't it just shut down and the process ends? Meaning that I can't call strace anymore? Do you have suggestions for that usecase? How would I know what the process id is going to be before I start the program?
