I'd been staying out of this conflict, partly because I'm not really in the know on WP Engine's behavior behind-the-scenes and, as weird as Mullenweg's plays have been, I don't like to comment on things I'm not fully read into.
But, this touches on a particular hobby horse of mine. It involves some old conflicts too, but I don't want to ruminate on them.
From about 2016 to 2019, I was heavily involved with trying to remedy what I considered an existential threat to the Internet: WordPress's auto-updater.
If that sounds alarming, consider the enormity of WordPress's market share. Millions of websites. W3Techs estimates it powers about 43% of websites whose server-side stack is detectable. At the time, it was a mere 33%.
For the longest time, the auto-updater would pull an update file from WordPress.org, and then install it. There was no code-signing of any form until I got involved. So if you pop one server, you get access to potentially millions.
Now imagine all of those webservers conscripted into a DDoS botnet.
Thus, existential threat to the Internet.
Eventually, we solved the immediate risk and then got into discussing the long tail of getting theme and plugin updates signed too.
You can read my ideas to solve this problem for WordPress (and the PHP ecosystem at large) here: https://gossamer.tools
Here's the part that delves into old drama: Mullenweg was so uncooperative that I wrote a critical piece called #StopMullware (a pun on "malware") due to his resistance to even commit to solving the damn problem. On my end, I reimplemented all of libsodium in pure PHP (and supported all the way back to 5.2.4 just to cater to WordPress's obsession with backwards compatibility to the lowest common denominator), and just needed them to be willing to review and accept patches. Even though I was shouldering as much of the work as I logically could, that wasn't enough for Matt. After he responded to my criticism, I took it down, since he committed in writing to actually solving the problem. (You can read his response at https://medium.com/@photomatt/wordpress-and-update-signing-5... if you care to.)
The reason I'm bringing this old conflict up isn't to reopen old wounds. It's that this specific tactic that Mullenweg employed would have been mitigated by solving the supply chain risk that I was so incandescent about in 2016.
(If you read my proposals from that era, you'll notice that I cared a lot about the developers controlling their keys, not WordPress.)
I don't keep up-to-date on Internet drama, so maybe someone already raised this point elsewhere. I just find it remarkable that the unappreciated work for WordPress/PHP I did over the years is relevant to Mullenweg's current clusterfuck. Incredible.
Since my knowledge on the background noise that preceded this public conflict is pretty much nil, I have no reason to believe WP Engine hold any sort of moral high ground. And I don't really care either way.
Rather, I'd like to extend an open invitation: If anyone is serious about leading the community to fork off WordPress, as I've heard in recent weeks, I'm happy to talk at length about my ideas for security enhancements and technical debt collection. If nothing else comes of this, I'd like to minimize the amount of pain experienced by the community built around WordPress, even if its leadership is frustrating and selfish.
Very interesting. I’ve been writing code for a while but if I’m honest I have no idea how code signing works. Any good resource on how it works especially in php?
That doesn't move the needle as far as restoring the trust you've broken.
You should negotiate with WP Engine to drop their suit contingent on your resignation. Maybe they'll go for it. Resigning is the only thing that would prove you're serious about allowing your power to be checked. And perhaps the only thing that would stop you from cutting a huge settlement check (probably within weeks and not the years you've anticipated).
Do you think that's something you're capable of? Do you care more about the future of WordPress and of open source than you do about your own power and rivalries? Will you prove it to us?
To be frank I don't believe you will. I'm pretty cynical about this kind of thing. But I've been wrong before. It would take a very strong person to admit, not just publicly but to their bitter rivals, that they had lost control and damaged their own life's work.
But if that person is you - it wouldn't be much, but you'd have my admiration.
---
Stark: Make peace with the Lannisters, you say? With the people who tried to murder my boy?
Baelish: We only make peace with our enemies, my lord.
> In this lawsuit against you and your mother, is it you or her who is accused of sexual harassment and racism?
Both (and the company through which they employed the plaintiff) are accused of the various discrimination, harassment, wage theft, etc., violations.
(EDIT: Though his mother is apparently accused of doing the direct racially- and religiously-bigoted statements, and the persistent graphic descriptions of Matt's sexual escapades, Matt's role -- other than as ultimately responsible as employer -- is participating directly in retaliation by taking complaints about the behavior back to his mother who accelerated rather than taking action to curtail them.)
> I don’t have access to read the case details.
You don't need access, you just need to go straight to the court site instead of a third-party aggregator.
And, if you had a nickel for every currently-active lawsuit against Matt and his mom for that kind of thing filed on June 9, 2022, you'd have two nickels, which isn't a lot, but it's interesting that there is more than one...
Am I reading this correctly? This guy owns an LLC through which he directly employs a personal healthcare team for his mother? And Mr. "Post-Economic" couldn't pay his nurses a fair wage?
> PLEASE TAKE NOTICE that Defendants hereby respectfully object to the Case Management Order, Notice of Time and Place of Trial and Trial Related Orders dated May 23, 2024. Since the filing of Defendants’ Case Management Conference Statement on May 21, 2024, Mira Hashmall, lead counsel for Defendants, has had a 5-7 day trial scheduled with a start date of March 17, 2025. With the trial in this matter starting on March 10, 2025, and trial in Case No. CGC-22-600095 starting on March 24, 2025, that would be three back-to-back trials and potential overlap amongst them.
In the years since I wrote that, several people have pointed out that "versioned protocols" are just a safe form of "crypto agility". However, when people say "crypto agility', they usually mean something like what JWT does.
> But if the protocol is not yet broken, then being agile isn't a concern, and if/when the protocol does become broken, then you can remove support for the broken protocol, which is what you'd be forced to do anyway, so a flexible approach just seems like a more gradual way to achieve that future transition.
This makes sense in situations where you have versioned protocols :)
This doesn't work if you're required to support RSA with PKCS1v1.5 padding until the heat death of the universe.
Hmmm, some recent protocols (thinking of MLS[1] here) have moved into a middle territory where they have a lot of options for piecing together a cryptographic suite, but then version that whole suite within the protocol. You can still change suites without changing the protocol, but it's not nearly as 'agile' as earlier suite and primitive negotiations.
Maybe something more like "cryptographic mobility" instead of "agility"? You can carefully decamp and move from one suite (versioned protocol) to another without changing all your software, but you're not negotiating algorithms and semantics on the fly.
> So I'm not so sure what's the point of encryption at rest in AWS except just to tick off a compliance and regulatory checklist.
> The private key is with them anyway, just don't encrypt and save few milliwatts of power.
"Them" is Amazon, a company with over 1 million employees, last I checked.
It's perfectly reasonable to trust the KMS team to keep your keys secure, even if you don't trust the RDS team to never try to look at your data.
I know it's tempting to think of all of AWS as a sort of "Dave" who wears multiple hats, but we're talking about a large company. Protecting against other parts of the same company is still a worthwhile and meaningful security control.
> It's perfectly reasonable to trust the KMS team to keep your keys secure, even if you don't trust the RDS team to never try to look at your data.
If the database is live, then the data is able to be decrypted and who knows where it ends up. Encryption at rest solves only the threat scenario where the RDS team has access to the database storage layer. It doesn't do anything to mitigate any threats after it has been read from storage.
As a customer, I don't know neither I do care how they have teamed up internally. Not my problem.
From my perspective, the secret keys I don't have. Just AWS has and they can decrypt whatever and whenever they want maybe because they have a warrant or some three letter agency has them do it.
> The author is confusing "it costs us nothing (now that encryption can be done in hardware and is integrated into most desktop operating systems) and protects in some scenarios, so yeah, we just decided to mandate it always be done" with "PEOPLE THINK ENCRYPTION AT REST IS A MAGIC BULLET LOOK AT ME I'M INSIGHTFUL, POST LINKS TO MY BLOG ON LINKEDIN!"
What in the article gave you that impression?
I do not hold this confusion in my mind, nor did I deliberately encode such a statement in my blog. I'm curious why you think this is what I was saying.
> The whole post is insulting to the intelligence of even a fairly junior desktop support technician.
If that was true, every time someone posts "Show HN: My Hot New Database Encryption Library in Haskell", they would be mitigating the confused deputy attack by design, rather than what we see today: Namely, failing to even protect against padding oracle attacks.
That's what the article was actually talking about.
If you're using a secure disk encryption technology, and you manage to clear the keys from the TPM or overwrite the header containing the KDF salts and other metadata, that should render the device data unrecoverable.
> I find it very hard to believe Amazon's or Google's servers do not already have full disk encryption.
I am confident that they do. Even better, they can be configured to use your KMS key rather than the service key, and you can configure KMS to use external key stores (i.e., an HSM in your datacenter outside of AWS's control, that you could theoretically pull the plug on at any time).
This is an interesting and important point that you raised.
> if you're storing the data on AWS and generating your keys on AWS and managing your keys with AWS ... you're doing it wrong.
This is a reasonable thing to do if you've decided that you trust AWS and expect any violations of this trust to be dealt with by the legal department.
It's less reasonable if you're concerned about AWS employees going rogue and somehow breaking the security of KMS without anyone else knowing.
It's even less reasonable to do this if you're concerned about AWS credentials being leaked or compromised, which in turn grants an attacker access to KMS (i.e., a government would be more successful by compelling IAM to grant access than they would trying to subpoena KMS for raw keys).
(Sure, you can audit access via CloudTrail, but that's a forensics tool, not a prevention tool.)
But that's kind of the point I wrote in the article, no? You need to know your threat model. You've stated yours succinctly, and I think it's a commendable one, but many enterprises are a bit more relaxed.
> It's less reasonable if you're concerned about AWS employees going rogue and somehow breaking the security of KMS without anyone else knowing.
That's the least of the concerns. Remember AWS is subject to court orders of all types (legitimate ones and NSLs). Even if nobody goes rogue, any data that AWS (or any cloud/SaaS provide) could access, must be assumed to be compromised.
> What's the best practice in managing user data keys so that data is available only when there's an authenticated user around?
What does it mean for an authenticated user to be "around"?
If you want a human to manually approve the decryption operations of the machine, but it can still store/encrypt new records, you can use HPKE so that only the person possessing the corresponding secret key can decipher the data.
At least, you can until a quantum computer is built.
A working definition for some apps could be: The user's data should not be available to the system if there isn't an active user session, such that the user's privacy interests are cryptographically protected in event of a breach or data leak occurring when the user is not actively using the system.
I wasn't thinking of manual approval of any cryptographic steps. Just that when you log in to work on your data stored in the system, the system can only then decrypt the data, and when you log out, the system forgets the keys until next time.
Okay, this sounds vaguely like a problem that may be solved by "HPKE where the secret key is reconstructed from a threshold secret sharing scheme" (>=2 of N shares needed, 1 held by the service and 1 held by the employee's hardware device, where 1 additional share is held in cold storage for break-glass reasons).
I would need to actually sit down and walk through the architecture, threat model, etc. to recommend anything specific. I'm not going to do that on a message board comment, because I probably am missing something.
> Obvious limitations is that you can't use the fields for sorting in queries (ORDER BY), and depending if deterministic encryption is not enabled, you can't use it in filters (WHERE) either. Same applies for any T-SQL logic on the data fields - because the encrypted blob is opaque to SQL Server - it is decrypted client-side. There is no workaround, except for pulling the data locally and sorting client-side.
But, this touches on a particular hobby horse of mine. It involves some old conflicts too, but I don't want to ruminate on them.
From about 2016 to 2019, I was heavily involved with trying to remedy what I considered an existential threat to the Internet: WordPress's auto-updater.
https://core.trac.wordpress.org/ticket/25052 + https://core.trac.wordpress.org/ticket/39309
If that sounds alarming, consider the enormity of WordPress's market share. Millions of websites. W3Techs estimates it powers about 43% of websites whose server-side stack is detectable. At the time, it was a mere 33%.
https://w3techs.com/technologies/overview/content_management
For the longest time, the auto-updater would pull an update file from WordPress.org, and then install it. There was no code-signing of any form until I got involved. So if you pop one server, you get access to potentially millions.
Now imagine all of those webservers conscripted into a DDoS botnet.
Thus, existential threat to the Internet.
Eventually, we solved the immediate risk and then got into discussing the long tail of getting theme and plugin updates signed too.
https://paragonie.com/blog/2019/05/wordpress-5-2-mitigating-...
https://core.trac.wordpress.org/ticket/49200
You can read my ideas to solve this problem for WordPress (and the PHP ecosystem at large) here: https://gossamer.tools
Here's the part that delves into old drama: Mullenweg was so uncooperative that I wrote a critical piece called #StopMullware (a pun on "malware") due to his resistance to even commit to solving the damn problem. On my end, I reimplemented all of libsodium in pure PHP (and supported all the way back to 5.2.4 just to cater to WordPress's obsession with backwards compatibility to the lowest common denominator), and just needed them to be willing to review and accept patches. Even though I was shouldering as much of the work as I logically could, that wasn't enough for Matt. After he responded to my criticism, I took it down, since he committed in writing to actually solving the problem. (You can read his response at https://medium.com/@photomatt/wordpress-and-update-signing-5... if you care to.)
The reason I'm bringing this old conflict up isn't to reopen old wounds. It's that this specific tactic that Mullenweg employed would have been mitigated by solving the supply chain risk that I was so incandescent about in 2016.
(If you read my proposals from that era, you'll notice that I cared a lot about the developers controlling their keys, not WordPress.)
I don't keep up-to-date on Internet drama, so maybe someone already raised this point elsewhere. I just find it remarkable that the unappreciated work for WordPress/PHP I did over the years is relevant to Mullenweg's current clusterfuck. Incredible.
Since my knowledge on the background noise that preceded this public conflict is pretty much nil, I have no reason to believe WP Engine hold any sort of moral high ground. And I don't really care either way.
Rather, I'd like to extend an open invitation: If anyone is serious about leading the community to fork off WordPress, as I've heard in recent weeks, I'm happy to talk at length about my ideas for security enhancements and technical debt collection. If nothing else comes of this, I'd like to minimize the amount of pain experienced by the community built around WordPress, even if its leadership is frustrating and selfish.