Yes, there's two frameworks in the kernel that now have some overlap: ftrace (incl. hist triggers) and perf_events (which can incl. BPF). But it's not a case where they magically got included. Initially they didn't really have overlap: ftrace was doing tracing, and perf_events was doing PMC stats. But they grew over time (especially perf) to the point where there was some overlap. This was noticed and has been discussed, and argued about. Eg, see the later half of https://lwn.net/Articles/442113/. At some point Steven Rostedt, ftrace author, had said:
> Now that perf has entered the tracing field, I would be happy to bring the two together. But we disagree on how to do that. I will not drop ftrace totally just to work on perf. There's too many users of ftrace that want enhancements, and I will still support that. The reason being is that I honestly do not believe that perf can do what these users want anytime in the near future (if at all). I will not abandon a successful project just because you feel that it is a fork.
Hist triggers is an enhancement to ftrace, not its own thing. BPF is a kernel facility originally designed for network I/O, that can also be used by ftrace and perf.
But yes, why not just one unified project? Such a project could bring in the experts from the other external tracing projects too (SystemTap, LTTng, ktap, sysdig, ...). Well, for better or for worse, Linux isn't a company, and Linus isn't its CEO. Scott McNealy, CEO of Sun while DTrace was developed, was fond of the saying:
If Linux was actually Linux Microsystems, the CEO could pick one tracing project and all teams would either toe the company line or leave. But we don't have that.
What we have, including external tracers, is a mess. There's too many tracers. And this becomes a tax on contributors who want to help fix the problem. Which one should you pick? Better lean the pros and cons of them all, which is no easy task.
Fortunately we're getting enhanced BPF, which does everything, so the tracing mess will finally have an end. Hist triggers was a surprise. It would have been a slam dunk before enhanced BPF existed, but now?? I suppose I understand it as an ftrace enhancement, and not its own thing.
That's not the way that Sun worked at all, actually -- and certainly not with respect to DTrace. To the contrary, Sun often deliberately allowed competing projects, believing that the internal competition would generate a better result. (Or, perhaps more likely, Sun simply did not have the command-and-control to prevent such internal competition.) So contrary to your assertion, there was never an executive fiat around DTrace (indeed, there was never much executive fiat at all at Sun), and DTrace had loads of internal competition -- but DTrace succeeded where the others didn't because it was actually useful for solving the problems that others weren't.
There's roughly ten different tracers for Linux. Yes, Sun liked competition, but it did not endure this situation like this -- what project had ten competitors, all staffed and paid for by Sun? If Linux were a company, I cannot believe the situation would have gotten this out of hand.
There were (at least!) ten different kernel debuggers at peak in the late 1990s -- and many different tracers too. (As the saying goes, you haven't heard of them for a reason.) My point is that Sun didn't get out of the morass of myriad half-assed solutions by executive fiat; it got out of it by a small, focused, self-motivated team that was hell-bent on building something that could work on real problems. The nascent success of DTrace had nothing to do with executive management, corporate heft, or marketing; these things only were brought to bear when DTrace had already succeeded where it mattered most: solving real problems.
A nice story, but it's not working out that way for Linux. The Linux kernel engineering talent is there, but split among the different tracers, and, have different priorities to work on. And this has dragged on for years. A rational company could see that it wasn't working (no nascent success), and then combine the right forces to get it done. Linux isn't a company.
You summed it up nicely, thanks. I kinda had LTTng and the other solutions in mind when I said there's many competing implementations, with varying degrees of functionality and maturity, and of course not all in mainline.
As a user, it's a mess because for example Erlang/OTP has added LTTng support, which I expect to be born out of a user requirement based in an environment like SLES or RHEL. That doesn't make it universally practical on a random kernel. Then we have DTrace emulation in SystemTap, and I've heard it kinda works, but I don't know how many probes exist for it.
If I want to trace on illumos or FreeBSD (NetBSD as well), there's just DTrace and I can share tracing code to some extent and the DTrace probes, say, in VMs have to be implemented once.
Like you, I hope there will be a coherent solution, but I don't see it yet.
> Now that perf has entered the tracing field, I would be happy to bring the two together. But we disagree on how to do that. I will not drop ftrace totally just to work on perf. There's too many users of ftrace that want enhancements, and I will still support that. The reason being is that I honestly do not believe that perf can do what these users want anytime in the near future (if at all). I will not abandon a successful project just because you feel that it is a fork.
Hist triggers is an enhancement to ftrace, not its own thing. BPF is a kernel facility originally designed for network I/O, that can also be used by ftrace and perf.
But yes, why not just one unified project? Such a project could bring in the experts from the other external tracing projects too (SystemTap, LTTng, ktap, sysdig, ...). Well, for better or for worse, Linux isn't a company, and Linus isn't its CEO. Scott McNealy, CEO of Sun while DTrace was developed, was fond of the saying:
> all the wood behind one arrow
For an April fool's prank, Sun staff put a giant wooden arrow through his office window. http://www.slideshare.net/brendangregg/from-dtrace-to-linux/...
If Linux was actually Linux Microsystems, the CEO could pick one tracing project and all teams would either toe the company line or leave. But we don't have that.
What we have, including external tracers, is a mess. There's too many tracers. And this becomes a tax on contributors who want to help fix the problem. Which one should you pick? Better lean the pros and cons of them all, which is no easy task.
Fortunately we're getting enhanced BPF, which does everything, so the tracing mess will finally have an end. Hist triggers was a surprise. It would have been a slam dunk before enhanced BPF existed, but now?? I suppose I understand it as an ftrace enhancement, and not its own thing.