For those of us coming from non-Linux systems (and I'm speaking for illumos/SmartOS here), the effort required to debug the kernel displayed here is galling. That this is so difficult reflects Torvalds' historic disposition against kernel developers[1]:
I don't like debuggers. Never have, probably never will. I use gdb all the
time, but I tend to use it not as a debugger, but as a disassembler on
steroids that you can program.
None of the arguments for a kernel debugger has touched me in the least.
And trust me, over the years I've heard quite a lot of them. In the end,
they tend to boil down to basically:
- it would be so much easier to do development, and we'd be able to add
new things faster.
And quite frankly, I don't care. I don't think kernel development should
be "easy". I do not condone single-stepping through code to find the bug.
I do not think that extra visibility into the system is necessarily a good
thing.
To be fair, this was a long time ago (17 years ago!), but the experience relayed here leaves one believing that Torvalds' historic attitude has cast a long shadow.
And if it needs to be said, Torvalds' arguments are themselves deeply confused; he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large. So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation -- both of which have proved absolutely essential for kernel development. (Indeed, it is likely that DTrace alone would have allowed the author to debug their problem, as its design center is exactly the kind of non-fatal failure described.)
The tutorial will certainly save others pain, but that such pain still exists at all in Linux is deeply unfortunate, and a vivid example of Linux not representing anything close to the state-of-the-art in systems development.
I agree! For Windows, getting symbols was trivial: you just pointed your debugger to the publicly-available symbol server[1]. You can see this attitude continuing in [2]: a beginner is warned away from using a debugger to understand the kernel behavior.
Wow, that second example is very telling. For whatever it's worth, before we started work on DTrace (ca. 2001), we prioritized a project to add not just symbol information but also debugging information to production kernels. This project -- the Compact C Type Format (CTF)[1][2] -- has proved essential many times over in the years since (and is a major reason why we can consider kernel debugging a part of the core system functionality). It is clear that a project like CTF is unlikely to even be understood let alone prioritized by those displaying such a dismissive attitude towards debugging.
Brian, I love your work but seriously. You've mispelled "read" as "understood" there and to write it off as a dismissive attitude toward debugging in toto is not something you should lower yourself to pretend to believe. Deep breath. DTrace is awesome, Oracle is not.
Yeah my kissing may not be up to scratch either... ;-)
The attitude towards debugging is dismissive -- and I'm not the only one to have drawn that conclusion. Indeed, it's the original author of this article who came across that and drew that inference -- one that I (obviously) share. To flip this around: do you think that the work outlined in the original article is work that should be expected of anyone wishing to debug the kernel?
I think there is room for reasonable people to disagree reasonably about most issues. Flip that around?
Linux is a long way from being the buggiest OS kernel I've ever used, how about you? Perhaps they've found and fixed some bugs? Perhaps they've prevented some from being written? Perhaps their approach is something that can be disagreed with, even strongly so, on the grounds of being less than optimal without suggesting that it has zero merit and by extension its proponents are somehow to be considered with derision? The inference that anyone hacking any OS kernel is too stupid to understand a differing idea is probably not necessary and unlikely to be justified in my humble opinion. You may of course reasonably disagree and maybe one of us did understand something the other did not on the point? Anyway this is now dull.
But don't be less opinionated, that would be the wrong response!
> Perhaps their approach is something that can be disagreed with, even strongly so, on the grounds of being less than optimal without suggesting that it has zero merit
If a yes or no attitude results in fewer bugfixes, all else equal, then that attitude does have exactly zero merit.
Errm, the email at https://lists.kernelnewbies.org/pipermail/kernelnewbies/2016... seems to be warning him against using an unusual set of custom compiler optimisation options that none of the kernel developers have ever tested with rather than against using a debugger full stop.
The beginner wants to disable optimizations, so he can better understand the kernel behavior by stepping through it in a debugger; he's told this isn't possible. Compare this with the availability of checked builds of Windows components, which support debugging by disabling optimizations: https://docs.microsoft.com/en-us/windows-hardware/drivers/de...
That's a great Torvalds emails which I haven't seen before! Here's the "point" though:
> I happen to believe that not having a kernel debugger forces people to think about their problem on a different level than with a debugger. I think that without a debugger, you don't get into that mindset where you know how it behaves, and then you fix it from there. Without a debugger, you tend to think about problems another way. You want to understand things on a different _level_.
I think that's really pretty reasonable. Surely you've seen a programmer who ran into a bug and submitted a pull request saying "I fixed it" but it's clearly just a messy work-around which misunderstands the problem and leaves a bunch of analogous problem cases unfixed.
I prefer printf-and-inspection debugging myself, I only reach for gdb once a week or so. I'm pretty diligent and my results are quite solid. It's fine that we have different styles, and because of well-designed interfaces, like process boundaries, we can co-exist in peace.
Linus is really counter-cultural in today's hip software world, where everyone says everything should be easier and easier. We just can't ever make threads or async or crypto easy enough. Meanwhile the popular applications and frameworks in use seem to get bigger and slower and buggier. Correlation, not causation, but still an annoying trend.
> he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large.
Where does he do this? The quote is only about single-step debugging.
> So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation
Do you have any evidence of this? It seems to me that dump traces and kernel probes have been an important part of Linux for many years.
No, the quote is not only about single-step debugging; his next sentence is: "I do not think that extra visibility into the system is necessarily a good thing." And if that still leaves too much ambiguity for you as to his thinking, read the rest of the screed -- and then remember how we got here: because it took someone two painful days to configure something that is considered core functionality on many other systems.
If the reason he's using a horribly unfriendly and difficult to debug API is because it's cross-platform and the more developer-friendly replacement is Linux only, then the fact that the API is so hostile certainly isn't proof - as bcantrill was claiming - that Linux is somehow going out of its way to be hostile to developers compared to competing OSes.
PF_KEYv2's difficult interface definitely isn't a Linux issue. But PF_KEYv2's limited error reporting means that understanding what's going wrong requires kernel debugging. The difficulty of kernel debugging is a Linux issue. ("I do not condone single-stepping through code to find the bug.")
Debugging a syscall or start of the process was a great way for me to learn the system internals. I have some experience with Windows debugging and, after reading the article, I find that configuring the kernel debugging in Windows is quite easy. And I really like the live kernel debugging feature, when you either use windbg (that requires the debug boot flag) or simply run livekd [1] to analyze the running system data (for instance ALPC connections, handles, or loaded drivers data). Is there anything similar available in Linux? I plan to learn Linux internals and would love to use the kernel debugger next to reading the source code and books.
Tangential, but if there is anyone interested in Windows debugging (including kernel debugging) have a look at the Inside Windows Debugging book by Tarik Soulami [2]
Great to see this and I hope I never have to do this on Linux.
A few years back, when I was finally getting my personal dev box off of windows, I took a very close look at FreeBSD or derivative. I hit the wall with poor touchpad support which made laptop difficult to use. I tried very hard to debug their touchpad kernel library but had no luck getting the results I needed. Anybody have links on how to do the same kind of remote kernel debugging on FreeBSD like above but using physical box rather than a VM?
I would still like to give that can another good kick'n :)
Two machines seems like overkill in this case - why not debug using User Mode Linux[1]? Debugging with two machines makes perfect sense if you are debugging a hardware driver, but not here.
Because UML cannot interact with most hardware and that is where bugs happen. Even VMs do not provide real hardware or allow you to bring down the machine with it.
Instead you can debug a crash using a trace and kexec dump. And also use quite fast ftrace infrastructure which is much better than plain old printf.
And if it needs to be said, Torvalds' arguments are themselves deeply confused; he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large. So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation -- both of which have proved absolutely essential for kernel development. (Indeed, it is likely that DTrace alone would have allowed the author to debug their problem, as its design center is exactly the kind of non-fatal failure described.)
The tutorial will certainly save others pain, but that such pain still exists at all in Linux is deeply unfortunate, and a vivid example of Linux not representing anything close to the state-of-the-art in systems development.
[1] https://lwn.net/2000/0914/a/lt-debugger.php3