Wow... reading this article in full really made me lose hope in OpenSSL, the project and the library.
I was well aware of the expected inconveniences any new major OpenSSL release would trigger (esp. older, less actively maintained applications having to adapt their API usage to keep working) going in, but what the linked github issue/PR comments hint at is just... mental.
As best illustrated by https://github.com/openssl/openssl/issues/20286#issuecomment... not only seem the core developers not care about runtime performance at all, they also seem to have a completely absurd perception of their own project, esp. in relation to other, genuinely small FOSS projects.
It's just wild. I hope this can still be turned around, and OpenSSL can somehow re-emerge from this clusterfuck as the stable bedrock of web transport security that we learned to rely on from 2014 onwards.
> I'm not aware of any accessible hardware that goes beyond a small to, maybe at a stretch, moderate scale. As a small OpenSource project, we have to rely on the community to do this.
That's especially absurd considering that it's possible to rent a VM in the public cloud with hundreds of CPU cores for the price of a cup of coffee per hour!
I've seen several projects where their learned helpless transforms over years into an obstinate point of pride.
For example, refusing to provide Windows builds in an era of free build pipelines in GitHub, virtual machines, cross-platform build tools, etc...
"We don't do that here!" -- says person who could trivially do that.
I don't know how it happens but sometimes very old OSS projects turn into those bikeshedding projects completely disconnected from reality. It only serves the self-fulfillment of the developers. I've recently ranted about libgd here in another comment. Definitely not as bad, definitely not as mission critical as openssl, but same symptoms.
That's two years ago and the issues have been fixed. Think of it this way: OpenSSL 3.0 was a release that added new, cleaner APIs and deprecated older, uglier APIs, so the focus was on that and not performance, but they've since put effort back into performance.
And by the way: OpenSSL has always cared a great deal about the performance of their cryptographic primitives, but the issue you linked to is NOT about that but about a performance degradation that happened in the code that _loads_ and references cryptographic algorithms.
IMO it's pretty lame to link to that issue as symptomatic of bigger problems in the OpenSSL community. And I say that as someone who was fairly involved in that issue as a user (I'm not an OpenSSL dev). It borders on the dishonest. How about giving them some accolades for responding to the issue, engaging the user community, and addressing the problem satisfactorily?
And as for them not having all the hardware that users run on, the perf issue in question did not require any particular hardware. The issue was systemic and easily reproduced.
Indeed, the OpenSSL devs ended up greatly improving how they do thread synchronization in general, because the issue was a combination of: a) using mutexes only, b) over time too many places got "get a reference to the cryptographic alg" calls that exacerbated all the problems with only using mutexes. So now OpenSSL has an RCU-like mechanism for thread synchronization that replaces mutexes for many read cases. That's _exactly_ the sort of large response program [to a systemic performance problem] that one would expect by an upstream that is well-funded and cares about performance.
So really, OpenSSL issue #20286 is demonstrative of all the opposite of what you say! It demonstrates that OpenSSL
- is well-funded,
- has skilled developers,
- is responsive to their users,
and
- is willing to make massive
changes to their codebase.
Show us how all of that is true of the others before you attack OpenSSL for having had a performance problem.
BTW, it was I who proposed switching to RCU for these synchronization problems, and I half-expected to be told that that's not portable. Instead and after convincing them that the problem was quite real they jumped wholeheartedly into developing an RCU-ish solution. How often does it happen to you that you point out a serious architectural problem to a popular upstream and then they take it seriously and solve it as quickly as possible? I'm sure it's a rare occurrence. It sure is for me.
I have yet to find a perfect community anywhere - open source or not. And, as a general rule, I agree that we should try to support open source communities with whatever path they choose to take.
In this case, it seems to me that OpenSSL has chosen to prioritize DX and broader accessibility over performance. Even as we have seen some performance improvements since the initial 3.0 release, it is still not where it was.
Thats ok.
As a widely used library, it may actually be a good idea for the project to prioritize the longtail of users over those who have the highest performance needs. This is a valid response.
The only issue I see is that this may not have been clearly thought out or communicated, which is a completely understandable oversight. There is another potential issue with rolling out an LTS release that was not "fully baked" but again - these things happen.
Overall, we should look to refine and understand the goals for the project, and be clear on the priorities of those leading and maintaining it.
> In this case, it seems to me that OpenSSL has chosen to prioritize DX and broader accessibility over performance. Even as we have seen some performance improvements since the initial 3.0 release, it is still not where it was.
Clearly you're wrong. They _chose_ to prioritize some things over performance for 3.0. They did not make the same choice for subsequent versions. Thus it's wrong to say that "OpenSSL has chosen" -- no, they've not! The choice you say they made was temporary. It wasn't even a conscious choice.
> The only issue I see is that this may not have been clearly thought out or communicated,
They did not make a conscious choice, and they did not stick to that choice later. You are twisting things.
> There is another potential issue with rolling out an LTS release that was not "fully baked"
Like everyone else, OpenSSL is moving to rapid release trains. Is 3.0 an LTS? Yes. Is 3.1 an LTS? No. Is 3.2 an LTS? Yes. Just upgrade and move on.
> Overall, we should look to refine and understand the goals for the project, and be clear on the priorities of those leading and maintaining it.
You're crying over spilt milk. The mess got cleaned up. The process was open and the team responsive. That instills confidence and defeats the FUD.
Actually, I'm trying to give the team the benefit of the doubt and assume good intentions. That may not be the case, but its a reasonable place to start.
And yes, they did choose to prioritize DX over performance - that was the major driving factor behind the re-architecture of the 3.x version. You stated this yourself:
"OpenSSL 3.0 was a release that added new, cleaner APIs and deprecated older, uglier APIs, so the focus was on that and not performance"
If you read the article, it is clear that while they have improved some of the worst performance issues, the core architectural problems (like being dynamic and poor multi-threading support).
In fact, that is the entire point of the article. Even if you look at OpenSSL's self-reported metrics, you can see that there are improvements from the 3.0 release, but still not up to the level of 1.1.1 or the other libraries tested.
Upgrading an LTS library is a problem for many orgs... which is why we have LTS versions in the first place. Suggesting it is not a problem just because you don't have a problem upgrading doesn't remove the problem.
Finally, you can see in the article that these points were brought up many times over many years, and even again after 3.0 was released, and the OpenSSL team was not responsive.
No crying over spilled milk here - simply trying to clarify that this is a problem for some users, and actually trying to support the OpenSSL team's decisions... even if they aren't the ones I made. AGain - assuming good intentions.
It does no one any good to blindly attack or blindly defend anyone. Lets be honest about the problems, honest about the issues, and honest about the choices the team has chosen to make.
OpenSSL was always the library developed by clowns, the only upside is it was used by everyone (so a sort of "can't be fired for buying IBM" situation). Big problem for libraries like WolfSSL, which is equally made by people that have a crypto hobby but doesn't have the distribution.
I was well aware of the expected inconveniences any new major OpenSSL release would trigger (esp. older, less actively maintained applications having to adapt their API usage to keep working) going in, but what the linked github issue/PR comments hint at is just... mental.
As best illustrated by https://github.com/openssl/openssl/issues/20286#issuecomment... not only seem the core developers not care about runtime performance at all, they also seem to have a completely absurd perception of their own project, esp. in relation to other, genuinely small FOSS projects.
It's just wild. I hope this can still be turned around, and OpenSSL can somehow re-emerge from this clusterfuck as the stable bedrock of web transport security that we learned to rely on from 2014 onwards.