I of course agree with all of this, but as someone pretty much at the bottom of the food chain who just wants to encrypt some data, there's often no libraries that safely glue the primitives together in the way that I require.
Granted, this is getting better, for example NaCl's crypto_box[0] is awesome and very hard to misuse.
But say you want forward secrecy now. chirp, tumbleweed.
I hope this doesn't come off as entitled, but I feel like the best way to get people to stop rolling their own crypto is to provide more/better libraries.
Say you have a requirement to build some chat somewhere, and it sais somewhere that it needs to be encrypted, which I think is a reasonably realistic example. So you use crypto_box, but then you realize you don't have forward secrecy. Ok, no problem, you remember libotr[1] just use that. Now you realize that OTR can't do group chats, so figure ok, most group chats are small everyone can just have an OTR session with everyone else and send their messages to everyone (which of course means you can give different people different messages, but let's ignore that). Next you realize people aren't always online at the same time and you can't send offline messages. So you remember Signal's double ratchet[2]. But libsignal was just built for signal and doesn't look easily integratable. Then I guess there's also (Meg)Olm[3] but that looks like the same story as signal.
I couldn't agree more, and it's actually something I've been working on for the last few months, though in a sense for the last few years. I was going to do a Show HN in the next few weeks, but heck, may as well mention it now too.
What I'm working on is called Etebase. Etebase makes it easy to build end-to-end encrypted applications by taking care of the encryption and its related challenges such as data-integrity, data-deduplication, collaboration and etc.
Everything is open-source, it uses libsodium[0] behind the scenes, and we plan on releasing a full spec soon as well as formally verify the protocol using Verifpal[1].
It's essentially a spin-off of the next version that powers my other project, EteSync[2] - a secure, end-to-end encrypted, and privacy respecting sync for your contacts, calendars and tasks.
i don't wanna sound dismissive, i'm sure Etebase is a good fit for some things. but GP was asking for a crypto library, not a whole SaaS that will host all their chat data
No worries, you don't sound dismissive, though let me address your points:
First of all, Etebase is not "a whole SaaS", since it's open-source you can self host it, so the SaaS is optional.
As for what the GP was asking, I disagree. I think it's exactly what he was asking for. He was lamenting the lack of complete solutions, and having to figure out things on your own. Without the server part and network protocol, Etebase is essentially just libsodium, which GP already mentioned.
I'm glad I found this, I've been spending my morning looking at libsodium and it's secure box function. I'm doing a refactoring of Socii, my social network project, and want to add E2E encrpytion.
Your pricing looks good and I don't expect to have users for quite some time, haha.
Please let me know (email: tom @ the domain) if you have any questions or feedback!
Though I'd love to hear what you are working on anyway (I love hearing about new privacy projects!) so would be great to chat regardless.
This looks pretty cool, but to be perfectly honest. At the moment I'd be quite weary to use this. And the reason is that I have no idea who you are and if you can write crypto code. Please don't take this as an offence, i'm sure you do, but I have no way of verifying this.
So my heuristic so far has basically been "is this from someone who's known to know their shit", e.g. djb and/or "has this been audited by someone/some company that's known to know their shit".
In general though, an end2end encrypted open source firebase-like stack does sound pretty awesome though. But you'd need to publish at least one external audit of the crypto for me to consider using it.
Anyways, I have it bookmarked and wish you success!
Thanks for the feedback, and these are all fair concerns. Some points that would hopefully alleviate some of them:
1. We have been running EteSync for >3.5 years and have been in the space for much longer, so I'm not just some rando, though yes, I'm definitely not djb. :)
2. We use libsodium for everything so the harder crypto parts are audited and battle-tested.
3. We are going to formally verify the protocol.
4. We have been been awarded a grant by NLnet's PET fund and as part of that we will get a mini security audit. Though to be honest, I think the formal verification is a much more serious assurance.
With that being said, I'm not trying to convince you. I think it's a very important point that we are working on addressing.
> Next you realize people aren't always online at the same time and you can't send offline messages. So you remember Signal's double ratchet
The part that enables sending messages to an offline person is uploading an ephemeral key and one time keys to the server.
The problem with libraries is many parts here aren't strictly cryptographic at all. So say one time keys. The server has keep track of them and your client also has to save them and keep track of them. Once in a while the client will have to replenish the keys on the server. It's hard to librarize this without being very imposing as to the implementation. And the actual cryptographic part isn't too different from normal DH schemes at all.
You should take a look at the library written by the author of this post (and the rest of my team), Tink [1]. It's written so that we can do security reviews at Google more easily, by making sure that the crypto library is fairly hard to misuse, in particular that the crypto library integrates well with key management.
>I of course agree with all of this, but as someone pretty much at the bottom of the food chain who just wants to encrypt some data, there's often no libraries that safely glue the primitives together in the way that I require.
>I hope this doesn't come off as entitled, but I feel like the best way to get people to stop rolling their own crypto is to provide more/better libraries.
Author here. I mentioned libsodium [1] and Tink [2]. We started Tink because we want to provide more/better libraries.
>Granted, this is getting better, for example NaCl's crypto_box[0] is awesome and very hard to misuse. But say you want forward secrecy now. chirp, tumbleweed.
It looks like you want to build an interactive protocol. I'm not sure if libsodium has a solution, but Tink doesn't. So far we've been focusing on encryption at rest. Can you tell me more about your use case?
> It looks like you want to build an interactive protocol. I'm not sure if libsodium has a solution, but Tink doesn't. So far we've been focusing on encryption at rest. Can you tell me more about your use case?
I don't have any particular plans of something I want to build at the moment. I was just using group chat as an example where there on one hand you're told to not roll your own crypto, but otoh you can't really just use someone else's crypto because there's no way to just use it.
Say I want to use an encrypted transport, that's trivial, I can just use TLS relatively easy. For the most straight forward case i can do `http.Get("https://example.org")` in go and not have to worry at all about the crypto.
If I want E2E, there's libsodium and tink, yes. But then am I "allowed" to build e.g. a forward secrecy scheme using ephemeral keys with these libraries? On one hand I know enough about crypto that I could do that, otoh, I also know enough about crypto that doing so would already make me uncomfortable.
So what I dream about is to have something like "ssh/tls" for E2E. Something like libsignal generalized. Of course you will have to do some key management, and it will never be as simple as `http.Get("https://example.org")`.
That only prevents attackers who read the current symmetric key from reading past messages. That won't protect you against the disclosure of a private key involved in the prior key exchange. https://doc.libsodium.org/key_exchange
The only way to achieve true forward secrecy is to throw away the private half of the involved key pairs. You need ephemeral keys. Which Libsodium does not provide out of the box. You'd need something like Noise. https://noiseprotocol.org/
I don't think you would ever want to choose a complicated protocol like Signal if you were not intimately familiar with how it worked. There would be a lot of ways for things to go wrong. If you understood the problem that well you are probably safer doing something simple targeted to your particular problem.
You don't need all the crypto features all the time.
Added: A good example based on the suggested need for forward secrecy. Forward secrecy only protects against the disclosure of a private key. In almost all cases, the ability to get a private key is going to mean that you also get any saved messages. So if your chat application requires the long term storage of old messages then you can skip the forward secrecy part.
It's more important property is keeping messages that are scooped up by malevolent 3 letter agencies secure if a random message in the middle is cryptographically broken.
Crypto is hard because you don't get quick feedback on whether you are doing well. The feedback will only come if somebody hacks it. You need to know what you are doing and most developers are not at that level. Most development happens by constantly iterating (write couple lines of code, find if it does what you expect it to do).
If you want to understand crypto imagine getting requirements and having to write an app. You can't run it before it goes to production. You need to understand every detail of the app is correct and you are not going to get any validation.
I like to think that crypto is not development. Development only happens after you did crypto (ie. writing down the process) and it is completely different from development.
I have rolled my own cryptogrophic protocol for transferring keys between devices that was subsequently approved by PCI. We needed the protocol in order to work around peculiarities of the particular HSMs we had and it used "funky" operations like encrypting block of data using 3DES in reverse (ie. using decrypting operation on plaintext block of data) so that it can be then digested by the HSM.
Very simple protocol, took a lot of effort to prove why we need it and to prove we know what we are doing.
>Crypto is hard because you don't get quick feedback on whether you are doing well.
Well said.
If you are to implement a sorting algorithm, you'll know immediately whether it works or is fast enough. Crypto doesn't provide this feedback. It's important to get help from others, if you can't tell yourself whether your code works.
Since any program using a cryptographic library is inherently now part of a cryptosystem, this is an incredibly dangerous point of view, because a) programs change, and b) cryptanalysis advances.
> took a lot of effort to prove why we need it and to prove we know what we are doing.
I hope this started by hiring people to prove that you didn’t, because the only path to trust is rejecting the null hypothesis.
What I meant is that cryptography is a separate activity that "works" in a different way. The result of "crypto" should be requirements for the software to meet.
These requirements should not be limited to description of how the algorithm works but also to important details like "no logging of any cryptographic material" or "duration of operations should be independent of data being processed or success of the operation".
The general problem with developer approach to crypto is that developers decide they have finished when their piece of code works. That just doesn't work with crypto (or security in general).
As to your second comment I can say that I was the only developer at the company and there was no option for hiring other people. As a developer I try to do the best I can with the situation I am in but CEO level decisions on hiring were generally outside my influence.
Whenever you work on something sensitive and potentially dangerous you generally get into a problem of "how I am going to convince them this is safe". In my case, having math background, the solution I decided to go with was to have a written rigorous proof of correctness of the solution and have two other mathematicians and PCI verify the proof (though I have no illusions PCI auditors went through entire document).
As far as cryptography and security goes it isn't possible to prove anything is safe and secure beyond questioning. I am happy in that I did about as much as I could.
For starters, for most applications there is no feedback because they are not popular enough for somebody to care to take a close look and publish results. Also, details regarding security are not usually made available for public which does not help. The protocol I worked on is used to bootstrap security for credit card terminals used for hundreds of millions of transactions and yet about 2 people ever looked at it really closely (my boss and another colleague mathematician).
For some applications you might get an outside opinion (audit or certification) but these are not typically aimed at checking if you did your job well, rather they check if you meet some kind of requirements. For PCI, for example, there is a questionnaire.
Also the feedback of this kind comes for really egregious mistakes, but the devil is usually in the details.
> Bad crypto code usually looks and produces results indistinguishable from good crypto code
This is such a weird part of cryptography... you can have two implementations, which both always produce the exact same output... and yet, one could be vulnerable because of side channel or timing attacks.
This is true for every piece of software, isn't it!? You can have 2 identical websites, one checks your username+password for real, the other always logs you in. Or even better, one stores bcrypted passwords, then other cleartext ones...
Use battle tested frameworks. For crypto, for sites, for everything.
This goes beyond that, though... in your example, the two programs DON’T give identical output with identical input; one will log you in with a bad password, one won’t. That is different behavior.
Likewise, with your second example, the storing clear text and bcrypted passwords are different outputs.
When I say the two pieces of code have identical outputs for identical inputs, I mean everything, not just what an end user sees.
Here is a toy example to show what I mean: say you have a function that takes a username and password, and compares it against a stored username and password (lets ignore the insecurity of storing a plain text password for this example). Now, you have two implementations of the function, and we know that both work as desired; they both will accept all correct username/password combos, and will reject all incorrect username/password combos. We have exhaustive tests proving that both return the same result when given the same input.
Normally, this is good enough - both functions are interchangeable, because they are guaranteed to always output the same data with the same input. You would probably choose the implementation that is faster, or used less memory, or whatever.
However, for crypto, this isn’t enough. Lets say in our exmaple, one implementation starts checking each bit of the password. If the first bit of the submitted password matches the first bit of the stored password, it moves to the next bit. If they don't match, it returns false because it knows that the passwords don't match. What a great optimization! You don't have to keep checking once you hit the first wrong bit. The second implementation checks every bit, even after it finds an incorrect match. So inefficient!
However, even though they both work perfectly in the sense that they will always return the correct result, the one that returns false on the first mismatched bit is vulnerable to a timing attack. An attacker can keep submitting passwords, and wait for the response to get a little bit slower... now they know they have the first bit correct. Repeat this for a while, and you can figure out all the bits.
This is obviously a toy example, but this technique has been used over and over again to break crypto implementations. You have to care about more than just the output when you deal with crypto!
> When I say the two pieces of code have identical outputs for identical inputs, I mean everything, not just what an end user sees.
But that's exactly the trouble with side channels: you've just defined "outputs" and "inputs" in a way that excludes them. If you widen your lens a bit to include, say, the timing of the code as an output, or even further to the physical changes of bits in hardware, then the outputs and inputs are no longer identical.
You can even extend that to the lock metaphor: when you pick a lock, you're using the tumbler itself as an "extra" input, and the feel of the pins as an "extra" output, instead of just the position of the pins. The tumbler and the feeling were there all along; it's not that they're really extra inputs or outputs, we just don't normally pay attention to them. Unless you take up lockpicking.
You're missing the point: in and out data via intended usage. Trying to move the goalposts afterward doesn't actually change what was intended. Consider ecesena's examples:
> 2 identical websites, one checks your username+password for real, the other always logs you in.
In that case, it's not a side channel. Presumably the intended use of authentication includes turning away bad credentials.
> one stores bcrypted passwords, then other cleartext ones...
That is an implementation detail. In itself, it's not a side channel because there's no way in the scenario to get the passwords. That cleartext implementation would appear at least as tough against timing attacks as bcrypt, in fact, because it would always decrypt all cleartext passwords in the same time - no info leak there.
I'm not. I get the distinction. The point I'm trying to make is that only considering the intended usage is an incorrect approach to design. The consequences of this constrained thinking can be particularly extreme in infosec, but it's just as true of software in general, or even, say, consumer products -- hence seeing all sorts of weird disclaimers on them these days.
Right, I understand that once you understand how side channels work that you can effectively program to avoid them.
My only point is that it is not something you normally have to deal with when you program. Normally, you are only worried about the inputs and outputs that you are intentionally interacting with. Side channels involve ones that you might not have even known existed.
I don’t think this is completely unique to crypto, there are plenty of cases in “normal” programming where you care about more things beyond inputs and outputs.
For example, two search algorithms give identical results, but one is O(n) and the other is O(n^2). They’ll both appear to work fine but one will catastrophically break down on certain inputs.
A similar one is where you have an O(1) hash table, but the hash function is vulnerable to collisions and sometimes explodes. That one starts to overlap with crypto in that it could be the basis of a DoS attack.
Besides running time, you might also care about memory usage, or designing an API that’s easy to use and robust against user error.
Related examples that come up in the web domain include:
(1) Zip bombs -- Do you accept compressed user content by first unzipping it and then examining the data? Good luck with the OOM.
(2) Catastrophic backtracking -- Are you still using regex to parse urls/emails? You're almost certainly using a library with worst-case exponential run-time rather than linear.
And so on. If the natural way to solve a problem is self-referencing you're almost certainly going to expose yourself to exponential memory/time consumption for malicious inputs.
That's a little different, because those websites are network services that are essentially black boxes to you. With cryptography, you can not only have two local programs, but you can have the source code for two programs and compile them yourself [1] but still have no clue if one is horribly flawed or not. At least without hiring seasoned cryptography experts to perform a rigorous analysis; and even then, who knows.
The same could be said when using a native application with an undiscovered use-after-free vulnerability or something, but in that case there are some theoretical inputs that'll cause the program to behave in a way you don't expect, like crashing or corrupting data, whereas with flawed cryptography the flaw may never give the slightest hint of its presence no matter how much or what kind of input you throw at it.
[1] For the sake of argument, let's say there's no risk of any kind of backdoors or "Reflections on Trusting Trust"-style hijinks
Not just that but something like AES using ECB is indistinguishable from random to the casual observer. Only when you’re aware of why ECB is bad will you understand what to look for to defeat it.
In case anyone wants to create a similar reminder for a specific image, a while ago I created a small utility that does just that: https://github.com/mhe/visualmoo
I think "casual observer" and "shove deliberately patterned data through the system then render it so that I can see if the patterns show through" aren't the same thing.
In the Tux data maybe 03038F12128F0303 turns into 49DC195026F3300A and that looks pretty random. It's only visible as the Tux image because every 03038F12128F0303 turns into 49DC195026F3300A, thus preserving the very bold overall pattern of the image.
If you shove a photograph of a kitten through it, that looks like noise, but Tux is made of these big bold shapes and so the resulting pattern is big enough to "show through" the ECB encryption.
You're basically implying that a casual observer would look at AES all by itself and then say things are good. And I think you're correct to imply that.
But that means that ECB isn't actually passing the smoke test, because the casual observer isn't analyzing AES+ECB. They're only analyzing AES.
I think when people say that a block cipher is indistinguishable from random they mean that for a given key k, the permutation 0 -> cipher(0, k), 1 -> cipher(1, k), ..., 2^n-1 -> cipher(2^n-1, k) where n is the block length in bits is indistinguishable from a random permutation of (0, 1, 2, ..., 2^n-1).
Actually a casual observer can distinguish ECB from random, though they may not realize that is what they are doing. For example if you manage to compress the ciphertext -- random data should not compress by much so if you can do any significant compression on ciphertext then you are distinguishing it from random bits. ECB should be compressable if the plaintext is compressable (though probably not as well as the plaintext do to the way ECB is divided into blocks).
Recognizing that this is a problem is another matter.
It's worth noting that, historically, cryptosystems have more often been broken through errors in key management than by actually cracking the encypherment algorithm. Key generation from an untrusted source.[1] Weak random number generation. Reusing keys. Keys intercepted in transit. Stuff like that.
I agree that modern cryptography isn't usually broken through design cryptanalysis, but I wouldn't bucket all the ways it usually fails into key management. There is a lot of misuse and misimplementation which goes beyond weak keys and key reuse.
I would rather say modern cryptography typically fails in one of three ways:
1. Incorrect implementation of low level primitives. For example: implementing textbook RSA (without padding).
2. Improper composition of correct primitives. For example: implementing an authenticated encryption construction using CBC mode with a MAC that shares its key with the CBC secret key, or simply using key = IV.
3. Incorrect usage of correct primitives. For example: nonce reuse with the same key, using regular cryptographic hash functions for password hashing, using an HMAC key larger than the block size, or using unauthenticated encryption to prevent tampering (e.g. URL parameters).
I cannot believe that we are talking about break-in cryptography and there have already been two XKCD posts but not the one about the reality of "cracking" cryptography:
So many ways to lose with cryptography. Cryptanalysis is a very alive field with very quiet buyers. So many websites use insecure cyphers for their TLS so many industrial systems have out of date cyphers or, worse, hand rolled "crypto" because they were written 20 years ago.
A protocol downgrade attack is easier than cryptanalysis, but it's observable. If all you want to do is surveil (not fabricate, modify, or deny) it's safer to do cryptanalysis since it's undetectable. Exfiltrating a key from, say, COMODO involves not just stealing the key, but using it involves generating a false certificate. It's a matter of scale for some actors. Crack as many cyphers as you can and log as much intercepted traffic as you can and decrypt it quietly. For important targets if that doesn't work, sure go hack the server and steal the key, certificate, or data directly.
Cryptography is a little bit like making fireworks.
For most people, and you are most people, you're better off to just buy fireworks made in a factory.
Follow the instructions, light them, they go bangity-boom-sparkle-puff, and everybody gets to finish up the evening with the same number of limbs and eyeballs as they started with.
Make your own, and you had better know what you are doing. Even if you do, Bad Shit is high on the list of possible outcomes.
Gunpowder may be a "low" explosive, but that doesn't mean it likes you.
Build up too much static and get a spark? Look around your workshop, and imagine your tools and furniture suddenly coming at you much faster than would be considered polite for inanimate objects.
Screw up the measurements? Get the fusing wrong? Now not just you, but also your guests, can experience the fun and excitement of being mortared.
Or you can be sane, and just buy some fireworks from one of those plywood booths that pop up every June.
“Ligh blue touch paper”, ok, got it. “ ... and stand well clear” yeah yeah, stupid safety advice. What kind of noob do you think I am? I’m a rockstar 10x ninja! <firework, caring little for egos, explodes as designed>
(Cambridge University is associated with a light blue colour, although exactly which light blue colour is controversial, its rival Oxford University is associated with a dark blue - and of course such security research may sometimes lead to fireworks of a different kind...)
What I don't like is when I hear people say the phrase "don't roll your own crypto!", I don't know whether they mean:
"don't use a cryptographic library yourself and build the cryptographic scheme yourself (HMAC, encryption, signing, etc.)"
or
"don't write your own cryptographic library and do all the base crypto functions yourself!"
I notice at the end you imply that a senior software engineer claiming they weren’t rolling their own crypto because they used OpenSSL was doing so in error. What sorts of issues could arise if they were using OpenSSL to handle cryptography? Or was it that they were trying to use low level APIs in OpenSSL to create some higher-level cryptosystem they perhaps didn’t fully understand?
For the very example-driven people, can you give a couple of examples of some of those non-obvious, subtle implementation details that completely destroy crypto schemes?
Symmetric encryption algorithms usually need an input called the initialization vector (IV), but different algorithms have different requirements.
AES-CBC requires unpredictability, but AES-CTR (and AES-GCM) require uniqueness. Misunderstanding of these requirements have led to real world attacks, see for example BEAST or https://eprint.iacr.org/2016/475.pdf.
Here's a random example. HMAC is frequently used in security protocols to generate more keys from a master key (for example, to generate temporary keys from a session key). The input to HMAC is (internally) padded with null bytes (<NUL>) before processing, so that for example "abc" and "abc<NUL><NUL><NUL>" will generate the same result. You may need to be aware of that padding when designing protocols that rely on HMAC, in order to avoid certain weaknesses. https://crypto.stackexchange.com/q/52161
As a very basic but very common example, I've seen many non-experts jump to encryption or symmetric crypto signatures whenever they want to prevent tampering with data, when what they needed was a simple integrity check.
Or, they want to do an integrity check, but don't realize that they need a canonical serialization algorithm for their implementation. So they use a standard, non-canonical serialization algorithm, and most of the time it works (because these algorithms typically don't vary that much, especially across tests on a single machine), but that's dangerous because it fails randomly in real life.
Encrypting too much data with the same key immediately comes to my mind. Due to subtle issues involving birthday bounds, encrypting e.g. terabytes of data using AES-128-CTR (or GCM or other counter modes) can be insecure. Note that this can become an issue when CTR mode is used as a PRG for other cryptosystems, particularly if there is no good entropy source available and the system is demanding a lot of pseudorandom bytes (e.g. because there are many TLS sessions being served).
Another issue to look out for is compressing plaintext before encryption, which in some cases makes the encryption itself close to useless. Skype had this issue a few years ago and it is a tricky problem for secure voice or video conferencing.
If speed was not an issue and you don't mind having to have 3 keys, could you cover all the bases by doing MAC-then-encrypt, then treat that as cleartext and do encrypt-and-MAC on that, and finally treat the result of that as cleartext and do encrypt-then-MAC on that to get the final thing you send?
Given how long side channels and timing attacks have been causing problems why haven't compilers tried to meet the requirements of crypto authors and implementors?
Feels like people are fighting against compiler optimisations more than ever.
Crypto code and non-crypto code have different requirements. For compiling non-crypto code you often want the fastest code that still matches the language spec. "Fast" is usually even relative to target platform. For crypto code, you need a very narrowly defined abstract machine that you program against and since that's a much stronger requirement than what the language is defined for, you take the next best thing by turning off all optimizations and hoping you understand well enough how your compiler generates code.
In other words, they already do their best by giving authors control over how much optimization is applied. Anything beyond that is a language design problem, not compiler construction problem.
This article is correct. However, it is important to call-out something that is far too prevalent, much code incorrectly uses correct crypto primitives. The world is full of calls to CBC with improper IVs, for one simple example. Another spot-on example mentioned in the article is timing oracles.
Security coding in general, not just crypto, uses an entirely different set of principles from functional programming. It is for this reason that "security" cannot be bolted-on, and cryptography cannot just be "dropped into" the code.
I am not a crypto maven, which means my ability to analyze the strength of various crypo systems is poor, although people say PGP is good at encrypting files and messages. I also had the thought of encrypting a file with one encryption scheme on top of another one - encrypting my file with one encryption scheme, and encrypting that encrypted file with another encryption scheme - although that idea is so obvious I am sure it is not a novel idea.
What is slightly novel is if I use a system with some tested (but uncertain) strength like PGP, and within it encrypt something with my own rolled crypto - this might protect me somewhat. Outside is the battle tested crypto of something like PGP. Inside is my own crypto system which might have some weaknesses, but which will probably take some hours of people working and analyzing it. Even if it can be cracked with 24 hours of analysis by skilled cryptographers, it still means it is not just spewed out of some computer somewhere as a trivially cracked message. Skilled cryptographers do not grow on trees, so my bet is that
#1 - Odds are something like PGP is not cracked, or at least a superpower can only crack a small percentage of messages a year in terms of computing power, so I am relying on it not being cracked or being hard enough to crack that my message is not deemed worthy
#2 - If it turns out PGP is quick and trivial to crack, I am relying on my message still not being deemed worthy enough to merit a few of the scarce labor hours of skilled cryptographers.
The consensus is that cryptography works. Even the NSA can't get past a properly encrypted message with PGP (assuming you use PGP correctly, and I've heard its UX is horrible). So, one layer of encryption is enough. No need to make your life more complicated.
The biggest fear is not that the crypto itself breaks, but that implementations don't actually implement the crypto correctly. At this point it's mostly a matter of being reasonably sure the software is bug free. Exacting, but not impossible, especially if it's kept simple.
"Modern E2E encryption is like sending and receiving messages with a top security truck, but then on arrival, storing them in a tent."
The biggest problem is security weaknesses at the end-point. Your selfie is travelling securely over the wire, and then it reaches the end-point device where there are typically 100's of unfixed vulnerabilities:
The consensus among cryptography engineers about PGP is not especially positive. And the consensus about "cascading" cryptosystems, like your own home-rolled system inside of, say, libsodium crypto_box, is overwhelmingly negative; don't do that.
I was expecting a kind of rant that one fills the reason why you want to create a new X. Then after fill the reasons why it wouldn't work. I have seen with calendars and other things here in hacker news. Someone remember the links?
I always wondered, since the NSA has intimate knowledge of the inner workings of AES, then can we presume they know its weaknesses, or have compromised it in some way? Excuse my naive question, I'm not a crypto nerd.
AES has been a major target of cryptanalysis, and it has held up pretty well. Publicly known key recovery attacks are only slightly faster than brute force, like [1]. There are some efficient related-key attacks, which might be a concern in certain applications, but not if keys are chosen randomly.
It's always possible that the NSA has found better attacks, but I don't think there's anything particularly suspicious about AES compared to other primitives. (For an example of a highly suspicious primitive, see Dual_EC_DRBG.)
We have no real way of knowing. AES is considered secure because, as far as is publicly known, it has sufficiently withstood attacks performed by many, many skilled cryptanalysts. As far as I am aware there is no reason to believe that AES provides anything less than its stated level of security. But again that isn’t to say it’s impossible.
My understanding is that such tricky side-channel free code ideally should not be directly written by humans. I know of at least one project that has made a lot of progress on this idea, although they say the crypto primitives are still at research level:
https://www.microsoft.com/en-us/research/blog/project-everes...
I'm not convinced it should even be written by machines. Even at an assembly level, removing side-channels depends heavily on implementation choices for a particular chip. A constant-time (for a given chip) binary targeting some instruction set very well might turn into a timing side channel when deployed on a chip with newer/different optimizations, especially with respect to memory/data dependencies.
I just use good primitives and glue them together as required in my application code. The only primitives that I feel safe playing with are: SHA2, AES, PBKDF2, CSPRNG, and Microsoft's DPAPI (for secret management). I always use first party implementations where possible. In this case, all of my crypto primitives are entirely written by Microsoft.
I never really screw with asymmetric crypto primitives directly. The closest I'll get to that area is stuff like loading a PFX byte array into an AspNetCore webapp for presenting a TLS cert.
Un fortunately, there are lots of things to specify in order to properly use the crypto you mentioned. It sounds as if you are already taking care of those things.
For people who may not be aware: Which SHA-2? Is it a truncated version? What encryption mode of AES? Are you using an IV? How is it generated? How is the key material protected? With PBKDF2, how many iterations, how was the salt generated, what PRF is used in it? Is DPAPI used in machine or user scope?
"glue them together as required in my application code"
This can easily be the source of problems. Bad compositions of crypto primitives are a common source of vulnerabilities in cryptosystems. A textbook example is MAC-then-ENCRYPT, which is not secure in general. Another common example is over-use of a cryptographic key, which can easily happen when underlying primitives are composed badly (asking for too many bytes from a PRG, or encrypting too much data in GCM mode, etc.).
"never really screw with asymmetric crypto primitives directly"
You might not be matching your crypto to your application requirements if you can write off a whole swath of primitives like that, or else you only deal with a very specific setting in which symmetric cryptography is sufficient.
Almost every crypto vulnerability I have ever found in production code came from systems designed with good primitives. It's not like we generally break systems by doing block cipher round function cryptanalysis.
I suck at crypto. Others are much better than I, and there's a huge plethora of really good implementations out there that get hammered by blackhats and red teams all the time.
There's too much at stake for me to add a naive implementation.
>How are people supposed to learn (from mistakes) if they don't roll their own crypto?
The right answer to this question, and not just on cryptography is: Stop it. There are billions of humans. They're already making plenty of mistakes, learn from those. Don't just make more and imagine that helps somehow.
If you feel the need to "learn from mistakes" you can find a healthy supply of detailed reports about mistakes other people already made and learn from them.
Most often what people actually mean is "I'm pretty sure I am uniquely brilliant and won't make mistakes" and I think the healthiest response is probably to laugh until you cry.
I watch a lot of cooking shows. America's Test Kitchen, Cook's Country, Good Eats, Milk Street, I watch Bon Apetite's YouTube channel, I read dozens cookbooks, all sorts of things.
Not a single one of them is a substitute for standing there in the kitchen and cooking. They help, they help a lot, but eventually you have to get hands on with something to actually gain experience with it.
Now, do you want someone to learn from their (incredibly common) mistakes on your production crypto system? No. Very Much No. But people learning from working on side projects and prototypes and whatnot? That's great, so long as they have a way to get feedback on what they're doing wrong.
But the thing is, there's no way to get feedback on what they're doing wrong.
If you make bad crypto in your prototype, then to you and most everyone else it will look just fine, so you'll get no feedback on misleading feedback. You won't get the chance to learn from your mistakes because you won't know what was a mistake.
You'll get realistic feedback only if someone skilled spends nontrivial time to analyze your solution (likely more time than it would take them to just implement a known good solution) - which will happen only if you hire them for an in-depth audit, or if you launch a production system where breaking your crypto will have a big enough impact to motivate someone to break it.
Every time I read comments like this, I'm left wondering the author thinks of the people who are actually doing the work in question - here it's crypto, in other cases it's new operating systems or game engines or programming languages or what have you.
This is what I think: those people are not some non-human species beyond the comprehension of mere mortals, nor did they crawl into the world already knowing everything in their fields.
We need more people experimenting and learning outside the boundaries of staid, safe product development, not less.
People actually doing good work get to make new and interesting mistakes. They hope not to of course, but that's just how it goes.
But what you and so many people want is permission to go on "experimenting and learning" by making the same mistakes the previous cohort of people "experimenting and learning" made, and those before them, and so on ad infinitum. That's futile. Stop it.
Since the horizon of your imagination seems to be computer software let me expand a little bit with an example from a quite different discipline which is entirely illustrative of my point.
Some years ago, an elderly couple were struck and seriously injured (one of them died I think) by a speeding tram. The accident investigation report found as you would expect a long list of causes, but one of them is particularly relevant to us here.
On the _mainline railways_ subject to oversight from the same authorities and investigations by the same team, it had for many years been the practice to require Compliance as a psychological trait of drivers. Compliant people are inclined to obey rules, all else being equal. If you tell a Compliant person that the speed limit is 45mph they will tend to go no faster than 45mph whereas people who are less compliant might ask "Why?" or just decide it seems safe to do 55mph instead since what could go wrong?
The driver of the tram was not tested for Compliance, his employers hadn't picked up on this requirement, and their drivers hadn't ever killed anybody by speeding before. Sure enough during the accident he was driving the tram faster than the maximum speed shown and was unable to stop it striking the elderly victims.
You don't need to make your own mistakes to learn from other people's. Stop it.
I thought I was making a rather obvious point, so here it is in plainer terms: you can't keep up a sufficient flow of new people in a field (especially a niche one) if everybody's response to "How do I start doing this?" is "No no no, this is only for a chosen few grandfathered-in greybeards and the rest of us are too stupid to even try". There's absolutely nothing wrong with people trying their hand at difficult things, even if the only thing it produces is first-hand experience/appreciation of how difficult they are. Black boxes and low bus factors don't aid progress, they hinder it.
Additionally, it's astonishingly not unheard of for people to, well, research the experiments and studies of those before them before trying their hand at their own; if you start all your own experiments by jumping headfirst in total ignorance, perhaps consider slowing down a little?
Then again you've chosen to go on a rant about speeding trams and fatalities when the literal third sentence in the linked article is "don't use [your crypto] in production until it's vetted by professionals".
There is actually a standard way for new people to get into crypto. However, it is emphatically not to write your own crypto and let others evaluate it. Everyone can make a crypto system they themselves cannot break. That says very little about the system's security, unless you're actually goods at breaking crypto.
That is why you begin learning crypto by breaking other people's crypto. Once you understand some of the pitfalls, your designs are far less likely to incorporate the same pitfalls.
> There is actually a standard way for new people to get into crypto...That is why you begin learning crypto by breaking other people's crypto.
And that is actual, useful advice for someone who is looking to get into crypto, not the equivalent of spraying an errant dog with a water bottle. Hence, like I said, studying and learning from others' experiments first.
To bring up game engine or OS dev again for instance, the answer to "I want to build my own game engine/OS" isn't "don't do that, you'll get it wrong", because people in those fields have actually put in the effort to create and document amateur-friendly resources targeted at various levels. And it's not like using your own OS or game engine in production is necessarily less dangerous than you using your own crypto in it, but again people in those fields somehow seem capable at grasping that people can [want to do] things without it even remotely being about a production service or product.
"No no no, this is only for a chosen few grandfathered-in greybeards and the rest of us are too stupid to even try".
This isn't even a straw man version of what I wrote. You should make your own thread to argue with this imaginary claim if that's what you want to do. Or better maybe start a Twitter thread about it.
As a reminder, here's that quote from the article, attributed to a co-worker but illustrative of a pretty common trope:
> How are people supposed to learn (from mistakes) if they don't roll their own crypto?
How are they supposed to learn? From other people's mistakes. There is no need to make those mistakes again yourself. Stop it.
Is that how you learned to walk? You observed other kids failing and falling constantly? Adults not falling? And you watched them until you got it and then you got up and have been walking flawlessly ever since?
Or were you thinking that you are uniquely brilliant, just tried it yourself for a while, falling and getting up, with an encouraging pat on the back by your parents every time you did, until you had figured it out?
You have correctly observed that children learning to walk is a frustrating process of trial and error. But weirdly, you seem to think this is an admirable property to be imitated.
How many say aeroplanes do you think the average airline pilot crashes during training? A few dozen? Maybe only a handful of big jets with passengers on them because presumably you'd practice stuff like "Not smashing an airliner full of people into a mountain" last?
No. We don't learn to fly aeroplanes this way because it'd be stupid. Aeroplane pilots learn from other people's mistakes instead. Even when they have an accident and they personally learn from that, every effort is made to also teach other pilots who didn't have the accident the same lessons. A guy will stand up and lecture other pilots about the time he did an Impossible Turn† and his point won't be "I'm fucking amazing" or "You should go do this and get yourself killed" but "Here's what I did wrong. Learn from my mistake don't make your own".
†Despite its name the Impossible Turn isn't actually impossible, it's just probably a bad idea and you almost certainly already did at least two things wrong if you're attempting it. It's an attempt to return to your takeoff runway by doing a 180° turn after engine failure in climb out. It probably won't work because you will consume too much of your residual energy. But sometimes people get lucky.
I don't really know that much about flying planes or learning to fly them, but I'm pretty sure that learning to fly planes involves lots of hands-on trial and error, mostly in small planes with an instructor next to them with another set of controls, and in simulators, in addition to reading about and listening to lectures on the theory of flight and past mistakes that have been made. They're still learning by doing, just in a structured environment where their mistakes don't actually lead to real disasters.
I don't think anybody would want to fly on a big passenger jet piloted by somebody who had never actually touched flight controls, just read books and listened to talks.
I believe that airline pilots also fly in simulators a fair amount, and that this gives them the opportunity to test their skills in a safe environment where failure is not penalised to the same degree.
funniest shit I've seen on the topic of doing your own crypto
Reply1
>The problem with Telegram and why it should be avoided is they tried to roll their own crypto called MTProto.
>That's is simply what you don't ever do. Encryption is done. There are many algorithms that are being tested and researched by experts, those are the algorithms to use. There is simply no need to roll your own.
Reply2
>The lead Telegram dev is a 3x International Math Olympiad gold medalist, won another gold in the informatiks olympiad, went on to earn two Ph.D.'s in algebraic geometry, all while working full-time as a programmer?
>Him rolling his own encryption algorithm is not the same as your copy-paste StackOverflow code monkey who scraped by with C's at his community college rearranging the alphabet letters in a caesar cipher.
> I'm gonna let you in on a little secret: Crypto is boring. We already know how to do it, it's been boring for like 20 years and unless you're doing something massively complicated like an anonymous cryptocurrency or you just make an enormous mistake, you basically can't mess it up.
I don't know what to make of this, except to say that the author is either misinformed or trolling.
While I appreciate the links to actual learning material, and though it is said in the nicest possible way, I get the usual "Crypto is hard, back off" vibe, without a clear path to victory. Sure it's hard, but it's not magic either. There are relatively few rules, that when followed give you a working, secure system. Those rules aren't easy to follow, but they are simple to know about. http://loup-vaillant.fr/articles/crypto-is-not-magic
About not using your crypto until it's been vetted by professionals… How do you get those vaunted professionals to even look at your work? For instance, I've been working on Monocypher for over 3 years, and the only way I could get professionals to actually look at it was to pay $7.000 (out of the OTF's pocket, I don't have the money). That's an unrealistic bar to meet, people need other ways to vet their work.
About CTF (Capture The Flag), and cryptopal challenges, my advice is: don't waste your time. The penetration testing approach to secure systems does not work. You whack a mole, two more appear. We need ways to prevent whole classes of errors, like proofs. For instance, tools like https://verifpal.com/ can be a great help when designing a protocol.
On crypto being deep and vast, it really depends what you're doing. While perhaps less than 50 people in the world are truly qualified to design a new cipher, that's not what people need when they "roll their own crypto". Most of the time they need a new implementation for an existing primitive (embedded programmers), or they need to invent a new protocol. Either is actually accessible to any professional programmer who actually follow the rules.
Even side effect circumvention. programming without the "if" statement is not that hard, especially if you stick to constant-time friendly primitives.
I wrote this article to encourage people to study the field I love. If I wanted to tell people to back off, why would I bother providing advice, material and telling people to have fun?
>Those rules aren't easy to follow, but they are simple to know about.
If the rules were so simple, how come your library had a signature bypass vulnerability that you did NOT understand its root cause until I explained?
The root cause looks deceptively simple, until you take your time to understand the underlying math. I didn't fully understand it until a professor at MIT explained it to me. Maybe it's just that I'm stupid, but it is never simple to me.
>About not using your crypto until it's been vetted by professionals… How do you get those vaunted professionals to even look at your work?
You can't expect people to pay attention to your work until you earn it. Why would anyone bother reviewing a random library from a random dude? Frank of libsodium fame didn't start by writing a brand new library from scratch, but built it on top of NaCl. If he did that, nobody would take him seriously either.
>About CTF (Capture The Flag), and cryptopal challenges, my advice is: don't waste your time. The penetration testing approach to secure systems does not work. You whack a mole, two more appear. We need ways to prevent whole classes of errors, like proofs. For instance, tools like https://verifpal.com/ can be a great help when designing a protocol.
This was how my friends and I got started. Maybe you were right that it isn't worth it, but it helps get us to where we are, being paid to do security and crypto.
There are two approaches to learning: top down or bottom up.
The former is what you get at university. If you want to learn crypto, you have to learn abstract algebra, linear algebra, probability theory, complexity theory, and computer security. If you want to learn complexity theory, you want to learn automata theory, computability theory, and algorithms. If you want to learn computer security, you want to learn computer architecture, operating systems, networking, etc.
The top down approach is systematic, but it might not prepare you for the real world. For that, you need internships and CTFs. They are reality checks with a fast feedback loop, showing you very quickly which skill or knowledge you are lacking or need improvement. They are also fun.
On the off chance that anyone here isn't familiar with 'cryptbe, he broke the Flickr URL signing scheme, he and Juliano Rizzo discovered and worked out the BEAST TLS attack (which, to hear Kenny Paterson describe it, more or less set the template for the next 10 years of applied TLS attack research), and then discovered CRIME, which is the first in a line of compression oracle attacks. He works on Daniel Bleichenbacher's team now doing Tink and and Wycheproof.
Hmm, that came out too harsh, sorry about that. I could clearly read your good intentions. It's just that at the same time, you bowed to the zeitgeist of "don't do this for real". It's kind of a ritual I see everywhere. Every time anyone writes about crypto, they feel obligated to say "oh by the way this is scary stuff".
Cryptography seems to be the only domain where this happens, even though many other kinds of code are just as critical: parsers, readers & players, network code… anything that reads potentially hostile input. I hate that double standard.
> If the rules were so simple, how come your library had a signature bypass vulnerability that you did NOT understand its root cause until I explained?
Because no single source actually explained what the rules were. I simply didn't know them. I didn't even know how to make a proper test suite when I introduced the vulnerability (in Spring 2017, well before v1.0.0).
By the time the vulnerability was discovered (in June 2018), I had a much better understanding of those rules, which allowed me to find the vulnerability from an odd report (which by itself wasn't a bug). The rule being "if you don't understand something, dig deeper". In other words, "don't mess with maths you don't understand".
Now however, after 4 years of practice, I think I have a rather keen understanding of the rules. And what do you know, they turned out to be fairly simple. Even side effects: you can know you've avoided all possible timing attacks. It's only a matter of making sure nothing flows from secrets to timings. (Side effects play a very small role in making code non-obvious. Almost negligible, compared to optimisations.)
---
Re CTF/challenges, my apologies. I should have qualified. I cannot deny they help you get a feel. I didn't go this route, but I reckon it's a valid one, especially if it's fun for you. Beyond that initial kick however, it really depends what you mean to do. Do you want to make crypto? Or do you want to break crypto?
If you want to break crypto (which would be extremely useful when dealing with existing systems), then sure, actually breaking broken crypto would be a huge help. Hands on experience.
I however want to make crypto. I don't care for legacy, I just want to either select or make something simple that suits my needs (something I'm currently doing at work, incidentally). And for this, I strongly believe that learning to break crypto is unnecessary. And beyond the very basics, inefficient. A faster way to build crypto is learning about testing and proofs. (I believe Collin Percival has a similar opinion https://www.daemonology.net/blog/2013-06-17-crypto-science-n...)
I don't think we need to go full top-down, however. My focus would be on mathematical proofs (proofs about algebra, about discrete maths, and about programs).
It's easy to write this when the safety of your users is both an abstraction and an externality; it's a lot tougher when you have to own your mistakes.
Please don't presume. I do care about my users. Their safety is not an abstraction.
You personally made sure it wouldn't be an externality either. I can count on you to make me pay for any mistake I make (only one of any consequence so far, and it was over 2 years ago: https://monocypher.org/quality-assurance/disclosures).
I know exactly how tough it is to own up to my mistakes.
He's someone I would really have liked to have around when I asked about the Edwards/Montgomery conversion for EdDSA all over the place. (Something I knew from the outset: when you deal with crypto, you don't do it alone. You seek advice and feedback.)
No one at the time warned me this conversion was dangerous. Yet another hint that we don't have enough cryptographic engineers.
Granted, this is getting better, for example NaCl's crypto_box[0] is awesome and very hard to misuse. But say you want forward secrecy now. chirp, tumbleweed.
I hope this doesn't come off as entitled, but I feel like the best way to get people to stop rolling their own crypto is to provide more/better libraries.
Say you have a requirement to build some chat somewhere, and it sais somewhere that it needs to be encrypted, which I think is a reasonably realistic example. So you use crypto_box, but then you realize you don't have forward secrecy. Ok, no problem, you remember libotr[1] just use that. Now you realize that OTR can't do group chats, so figure ok, most group chats are small everyone can just have an OTR session with everyone else and send their messages to everyone (which of course means you can give different people different messages, but let's ignore that). Next you realize people aren't always online at the same time and you can't send offline messages. So you remember Signal's double ratchet[2]. But libsignal was just built for signal and doesn't look easily integratable. Then I guess there's also (Meg)Olm[3] but that looks like the same story as signal.
[0] https://nacl.cr.yp.to/box.html
[1] https://otr.cypherpunks.ca/
[2] https://en.wikipedia.org/wiki/Double_Ratchet_Algorithm
[3] https://gitlab.matrix.org/matrix-org/olm/-/tree/master