Hacker News new | past | comments | ask | show | jobs | submit login
Finding Java Thread Leaks with JDK Flight Recorder and a Bit of SQL (morling.dev)
197 points by gunnarmorling on March 4, 2023 | hide | past | favorite | 58 comments



JFR is one of my favourite Java features. Like eBPF but with more events baked into the runtime and easier to use.

Gunnar has used it in some cool ways. Not just https://github.com/moditect/jfr-analytics, which he showcases in this post, but also https://github.com/moditect/jfrunit


JFR is just awesome. Thanks a lot for the nice words, Ron! Means a lot to me, coming from you.


And if my history recollection is right, originally born on J/Rockit JVM.


Correct.


Say what you will about Java, but its tooling is not lacking.


One thing I love about the JVM is that the tooling can be used in production if necessary- generating thread dumps, JFR recordings, monitoring heap usage, etc., all of this is invaluable when diagnosing problems.


Yeah, if needs must, you can even attach a remote debugger to a process i prod. Pretty crazy.


Yes! I’ve had to do this at a previous company, while leadership was demoing at a conference, there was a bug that would crash the system. Connected remotely through a debugger and listened for the condition, modifying state to keep the crash from happening. Can’t remember the details, but it was exciting.


i miss the tooling so much, theres not many programming languages that can compete, maybe c# with the jetbrains suite


.NET with Visual Studio Enterprise. There is enough of .NET that Rider still doesn't do.


Any examples? The only one I've encountered in the last few years is editing .net core WinForms UIs with the visual editor. In every other use case of mine, Rider has outclassed Visual Studio.


WPF, Blend, hot code reloading, repl, mixed language debugging with C++/CLI, C++/CX, C++/WinRT, SharePoint, Dynamics, SQL Server SP, Architecture tooling, coverage and automatic unit tests, COM interop tooling, Azure workloads integrations, some of the examples that come to my mind.

Also, when already paid for MSDN license which includes VS, why pay for Rider?


> paid for MSDN license

I believe a lot of companies are not directly paying for MSDN licenses, but retrieve them for free through a Microsoft partnership and by fulfilling conditions to obtain silver and gold competencies. I think this is the case for a lot of smaller companies.

Microsoft however is changing their partnership conditions by focusing more and more on Azure. To be able to stay a partner with similar benefits, you basically need to generate more and more revenue on Azure for Microsoft. Something that will not be doable for a lot of smaller companies in my opinion. And this also depends a lot on the focus of the company, whether it is a pure software development shop or also a reseller of Azure services.

Besides that, .net development is no longer necessarily windows development. My current team consists of 6 people. 1 uses Linux, 2 use OS X, 3 use Windows. We use Rider, Webstorm and Datagrip. We develop .net backends that run on a Linux app service plan. The front end is in Angular. The database is postgresql.

A lot of the .net developers in our company asked to use Rider instead of the VS they get through an MSDN license. Most of them are Windows users. Some people clearly think Rider is superior to VS, myself included…


As polyglot agency, we use Java for UNIX workloads, .NET is having its Python 2 / 3 moment with .NET Framework / Core, and plenty of stuff in large enterprises will probably never migrate. It is no accident that Microsoft is doing all the blog posts about how internal teams are migrating, or eventually started coming up with porting tools, which in the beggining they refused to provide (like in WCF's case). [0]

And even so, plenty of Microsoft products like Dynamics, SharePoint, VS, SQL Server CLR, are yet to make the transition to .NET Core.

So when doing UNIX workloads, we rather use languages what were created and grew up on that environment.

Visual Studio for .NET, Eclipse for Java, VSCode for node/devops (any of them for C++ workloads if needed as well), are basically how things go over here.

[0] - To be fair, Java is also having its Python 2 / 3 moment with anything beyond Java 8. Finally got to deploy Java 11 LTS into production (hurray!).


> plenty of stuff in large enterprises will probably never migrate

I think you are right. I worked, and still assist, on a couple of big projects that are built on .NET Framework. I recently pushed to have those upgraded to the latest .NET Framework version, latest version of their libraries, and have the csproj-files upgraded to the new style. Those projects are all WCF, they use NetTcp, they use transaction flow over WCF, some use WCF-msmq. There is a big dependency on distributed transactions over service boundaries, sql server and queues, and on WCF. It would be a major effort to rewrite those to run on .NET Core without distributed transactions and over HTTP APIs.

Those systems have been running just fine for over 10 years, are deployed on-prem and are only used internally on the company network. As long as .NET Framework is supported, there is no value in a rewrite to .NET Core for the business.

In the company where I work, we actually do C# on .NET Framework (maintenance, support) + Core (all new projects), Java, Delphi, and Angular + TypeScript for frontends. We use a lot of the JetBrains tools: IntelliJ IDEA for Java development, WebStorm for frontend development. We used to use Visual Studio + ReSharper for C# development and since Rider came out, a lot of people moved to that.


That’s what you get if something as basic as debugging is not open-source for your platform.


Rider has a great WPF editor (again, I would say it often surpasses VS in terms of code completion), and is capable of hot code reloading (for server code - it doesn't work with WPF AFAIK).

I'm also not sure what you mean by "REPL" as I tend to think of that as a language feature, not an IDE feature (most read-eval-print-loops are done through a terminal, which Rider has).

Rider also has coverage and automatic unit tests (and it's sibling product Resharper had it for years before Visual Studio added it).

"Architecture tooling" could be a quite broad category, but as far as where I've used Jetbrains products in this sense, they can automatically build class diagrams and the like from your code, and even export them to PlantUML if you want.

Jetbrains also offers an IDE that can handle C++ just fine - https://www.jetbrains.com/clion/. I don't know how well it handles WinRT, but I'll admit I've never had the need to develop or deploy a WinRT application. I do know that they've started building in MAUI integration [2].

I'll confess I don't use Dynamics, or know what SQL Server SP is, but Rider has great Database tooling built in. I've never had a problem connecting to MS SQL Server, SQLite, Postgres, or MySQL databases with it.

> Also, when already paid for MSDN license which includes VS, why pay for Rider?

I guess this question is aggressively asking why someone would choose Rider if they already have Visual Studio Enterprise, and beyond it's great tooling, the main benefits for me are the great code completion, the snappiness of the UI (you can use the editor before it has loaded the AST), and how rare crashes are with it (and it has seemingly gotten even rarer in the last few months, I can't remember it crashing). Beyond that, if you have to choose between the two, an Enterprise MSDN license is $6k/yr [0], and a commercial Rider license + the other Jetbrains C# tools is $470/yr [1]. The former is usually a difficult question with your manager and/or purchasing, while the latter hardly raises an eyebrow.

[0] https://www.microsoft.com/en-us/d/visual-studio-enterprise-s...

[1] https://www.jetbrains.com/rider/buy/#commercial

[2] https://blog.jetbrains.com/dotnet/2022/08/02/rider-2022-2-re...


> The former is usually a difficult question with your manager and/or purchasing, while the latter hardly raises an eyebrow.

The one thing is that everyone on your team probably uses Visual Studio or has a license if you’re a .NET shop, that may not be true of Rider or ReSharper. Your manager is already convinced of the need for VS. (I just bought my own JetBrains Toolbox because I had a coupon, but my company will pay for ReSharper if we want it)


Instead of VS, buy two IDEs on top of MSDN, great.

SQL Server Stored Procedures, which can be written in .NET, as SQL Server hosts the CLR. This includes the whole debugging and deployment stuff.

Architecture tooling means Enterprise Architect level not basic PlantUML stuff.

MSDN licenses are always part of Windows development workshops, the only question is choosing between professional or enterprise.

So anything JetBrains is always extra.


I don’t know about .NET, but REPLs don’t need to have anything to do with terminals. One example of a REPL are Swift Playgrounds in XCode. Jupyter Notebooks are also a kind of REPL.


Certainly, and there are numerous REPL environments (LINQ Pad springs to mind), including via the console.


Rider does have parallel stacks, but does not have a function to show tasks. Some deadlocks are much more difficult to debug if you can't show the tasks.


JDB is pretty terrible tbh. Granted most of its use is indirectly through IDES, but still.

Also, having tooling to address/fix the many issues Java has isn't really that good in the first place.


> and the stack trace of the parent thread when starting the child thread

That's incredibly useful. We have a strict policy of all threads and pools being named, but sometimes default named threads sneak in making it difficult to figure out where the thread is used for.


Java is not the most appealing programming language, but you've got to love the JVM, the standard library, all the libraries and frameworks, plus the tooling around the ecosystem.


Here's a real-world example of using C# to create a set of file paths, aiming to eliminate duplicates. I'd've written this in Java as:

    Set<File> s = new HashSet<File>();

    s.add( new File( "c:/temp" ) );
    s.add( new File( "c:/temp" ) );

    System.out.println( s.size() );
This prints 1.

The C# code:

    ISet<FileInfo> s = new HashSet<FileInfo>();

    s.Add( new FileInfo( "c:/temp" ) );
    s.Add( new FileInfo( "c:/temp" ) );

    Console.WriteLine( s.Count );
This prints 2.

They are line-for-line identical. IMO, Java prints the correct result. C# may have reasons for this behaviour, but when you're looking at the code, it's not obvious that two FileInfo objects for the same paths are not equal.

When I wrote KeenWrite[0], I chose Java because of all the reasons you mentioned, plus JavaFX. JavaFX is one of the richest cross-platform GUI libraries available for desktop applications, although it does have quirks.

[0]: https://github.com/DaveJarvis/keenwrite


I don't understand the point you're trying to make with your example. Is the concern here that in C# `FileInfo` does not override `Equals` and because this is inconsistent with other standard lib objects this is a trap for developers? If that's the case, well, Java has its own share of weird and inconsistent behaviour that developers simply need to be aware of.

---

On a tangent, is there a reason why FileInfo does not override `Equals`?


FileInfo is a stateful reference type, so overriding Equals and GetHashCode to use value equality would be a Bad Idea™.

E.g.:

    var f = new FileInfo("image.jpg");
    var set = new HashSet<FileInfo> { f };
    f.MoveTo("image2.jpg");
    Console.WriteLine(set.Contains(f)); // Would be false (sometimes)


Then FileInfo should not be hashable, so a Set of it could not even be declared.


But the current implementation using reference equality is perfectly valid, and often quite useful.


https://en.wikipedia.org/wiki/Principle_of_least_astonishmen...

Here's another:

    string path = "/tmp/filename.txt";
    FileInfo f = new FileInfo( path );

    File.CreateText( path ).Dispose();
    Assert.True( f.Exists );

    f.Delete();
    Assert.False( f.Exists );
Astonishingly, the test fails because the result of Exists is cached. ¯\_(ツ)_/¯

My expectation is that if I put two things that are identical---for all intents and purposes---into a set, only one copy is added to the set. Likewise, upon deleting a file, I expect that testing for its existence always returns false. Both of these expectations are based on my mental models for math and file systems.

I haven't tested the behaviour of caching file states in Java, but I suspect it will return false if the file is deleted. Point being, in my limited time with C#, I've found it to violate the principle of least astonishment in ways that Java does not.


C# FileInfo uses reference equality, because that's the default for objects and they didn't override it. Why? I suppose they thought of it as a helper to get information about a file (or open it), rather than being a file handle itself. Or perhaps it wasn't clear what value equality meant for FileInfo. If two FileInfo point to the same file, but use different paths (relative paths, case insensitivity, hard links, etc.) should they compare as equal? Java just string-compares the two paths, but that's not what I'd want it to do. (I wouldn't want reference equality either. I'm not a big fan of reference equality, personally, and C# has gotten more value-equality oriented over time.)

Doesn't Java also have things that use reference equality? Arrays?


>Doesn't Java also have things that use reference equality? Arrays?

Yes. There are inconsistencies all over the place. In fact, OPs example of java.io.File is ironic, because that object had a number of issues and was somewhat replaced by java.nio.file.Path [1].

Even outside of that, file equality in general is fraught with a lot of subtlety. For example, should java.io.File equality be case sensitive or insensitive?

[1]https://docs.oracle.com/javase/tutorial/essential/io/legacy....


Perfectly intuitive, just like how `"foo" + "bar" == "foobar"` is only sometimes true.


I aimed to eliminate duplicates of URLs using that approach in Java. Do they resolve to the same IP? Same URL.


Yeah, URL is from the first release of Java and should be avoided. You want to use URI instead.


Hence, Kotlin.


I've a couple of years of experience of using Kotlin. In many ways I loved it, and it would have become my JVM language of choice but over time I've started feeling uncomfortable about recommending it.

The language is completely closed and controlled by a single commercial software company and there is no community process involved in its evolution and design. I had believed that this business model of programming language design/ownership died decades ago when everyone abandoned all the bespoke commercial languages/IDEs of the 1980s and early 1990s. The way Jetbrains have scuppered the attempt to provide a reasonable vscode plugin feels like reliving the 1990s where the single company providing the language spec, compilers, IDE and tooling for "their" language and aggressively prevented others from providing alternative tools and compilers.

Also compile times are horrendous - even with modern hardware, the compiler barely seems to be able to manage much more than a few 1000 lines/sec. Again, my age means I remember decades ago when compilers/IDEs advertised their performance in terms of 100,000s of lines per second. This means edit/compile/debug workflow is very painful once your project grows to non-toy sizes.

I also grew to dislike the ethos of the "community" which basically seemed to involve condescension and passive aggressive responses to to any questions regarding the design of language features particularly by Jetbrains employees.

Kotlin is undoubtably a more "modern" feeling of development than Java (or C++ for example) but I'm not sure such vendor lock-in is worth it given many other languages are catching up (including Java) or exceed it. Also I used C# for about a year subsequently and realised that Kotlin stole most of its best features from C#. Languages like Rust, Zig, Swift, etc. are just as "modern".


It's also a bit all over the place. Kotlin seems to be afraid that Java is catching up and so is branching out to Multiplatform.

All in all there are less resources on Kotlin but too many things they want to do. Not only are they building the language but they have to reinvent everything because they don't work on Multiplatform.

It used to be simple - a better Java (so it claims). Now I'm not so sure what the direction is.


Or Scala, Groovy, Clojure, Ceylon.. we have heard it a few times already.


So when are we getting Kotlin Virtual Machine, designed for Kotlin?


Does the JVM impose heavy constraints on Kotlin?

I ask because the JVM is an amazing bit of technology that has been battle tested for decades. Would be hard to beat I would think.


One of the major issues that Kotlin faces is that it’s hard for them to change the VM to provide a feature they want while Java and the OpenJDK developers have the freedom to either change the VM or the language to provide something.


I don't think it's too burdensome, because they can emulate features until the JVM gets support. I can think of several:

- Data classes can become JVM records with the @JvmRecord annotation and some additional type constraints.

- Inline classes will certainly become Valhalla value types (hence the required @JvmInline annotation).

- Nullable types are supported with compile-time checks and additional runtime checks inserted by the compiler at time-of-use, and if Valhalla adds a nullability marker to the bytecode, Kotlin can support it.

- Coroutines/structured-concurrency already uses Executors under the hood, so it's likely that we'll have Loom thread support without any changes to the coroutines APIs (though the standard dispatchers will need to use virtual threads themselves). Coroutines themselves are CPS-transforms of ordinary code and don't need VM support.

- Tail-call elimination is supported by transforming to trampolines, something which the JVM will probably never support until Java needs it.


Right but, in my opinion, those all come with ergonomic issues from not being able to change the VM. In general the quality of the feature provided by Java will be better because of their ability to mold the VM to the language.

For example syntactic cooperative coroutines vs preemptive coroutines provided by the VM. Kotlin is now stuck with the colored function problem and has the stack trace issues that the Java team is able to avoid by not only being able to change the language.

That being said, I think they're both fine languages with different goals.


I'm not convinced colored functions are a problem, personally. You either color an async function with a Continuation argument and place yields, or you can mark it as a CPS function with a keyword, so it's not like coloring goes away by omitting the keyword. And making async pre-emptive introduces a different kind of complexity that can be just as hard, if not harder, to reason about for anything more complex than "park this thread for IO".


Sounds like Kotlin and other JVM languages need greater representation in OpenJDK then, if that would be possible.


I need to learn to use jfr apparently. Just something I’ve known existed and never used.

We attach a hawtio.io jvm agent to every single prod jvm we run and query the agent using telegraph. (If you don’t do this by the way, you should!) I noticed in the GUI portion of hawtio, it slows you to start/stop a jfr recording, but I’ve never messed with it


HN being very anti Java, why does such many Java articles make it to the front page?


Maybe the commenters are "anti" and the voters aren't (more comments being also a detriment to getting to the frontpage)?

But I don't see any real "anti" comments, only valid criticism and I say that as a Java dev myself. It does have its wrinkles. But for it's age and popularity it's still a pretty good language.


A lot of it is misdirected hate.

In the first dot.com boom, a lot of people worked for tech companies, and Microsoft was the Google|Apple back then. Their corporate culture was dog-eat-dog nonsense, but it trickled down through the industry... and of course at the time Java was super super popular, even though _it really sucked_ back then (not so much now, it's quite awesome).

One of the most _hated_ positions was "The Architect" who usually rose to the top via floating on top of a bubble of attrition; so rather than being promoted because you're good at leadership or a great person that's fun to be around or a technically savvy and loved to teach... these architects were ignorant egotistical maniacal dictators that got promoted merely because everyone else quit. They turned static analysis tools into weapons of war. They made arbitrary standards that really didn't work for anyone, rather than trying to work for someone. Everything had to be over-engineered (sound familiar? If not, look at today's SPAs) with AbstractFactoryProducerMethodProducerImplInterfaces. I worked at a lot of these companies, and it just wasn't a fun time to be in software development.

Rather than people seeing the reality that Poor Leadership and zero charisma as the root cause, somehow people arrived at Java was the core problem.

A lot of the things were good in their intention; for instance, having the whole company use tabs instead of spaces (or vice versa) and having a universal code formatter have the potential to increase communication. But instead, it was just executed poorly at nearly every shop and the vast majority of the problems got turned into Dilbert comics.

But yeah, Java.


That's not my experience at all, if anything I feel like HN is quite fond of Java. The anti Java crowd is mostly on Reddit instead where it's the subject of memes and jokes and the like.


Except /r/java where Java is God


I think it's safe to assume that as a general rule on /r/<X>, <X> is god.


There are lots of people in /r/java that defy God :p


There's a release every 6 months which have been packed with great stuff, and don't forget half the world's economy still runs on Java.


JFR is nice but I can't seem to find issues as easily with the new JMC as I used to 5.5 or before. I'm not sure what I'm doing wrong though.


For a split second there, I thought a post-airline accident flight recorder had some Java heap dumps that showed major issues and thread leaks.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: