How do you decide that a piece of software is "well-written"? Even very strict review processes missed critical issues... and none of those are team for rare software.

Instead, the proper advice is to reduce attack surface, not rely on some allegedly obscure piece of software.

Using results from academia is often supremely impractical. The actual software and designs are often unavailable or perhaps impossible, very definitely unverified.

Your homegrown CTR crypto might expose you for a related key attack. How do you know, since nobody targets it, until it is broken?

Guessing target architecture given an exploitable but is trivial. You need a simplest of data leaks. There aren't many options available either. You can use x86, MIPS or arm, the latter two in big endian or little endian. Other much more unlikely targets can be POWER or ia64. Custom micros can be PIC or MCP51, maybe Atmel. That gives only a bunch of options to try out.

You already have to tangle with much stronger security measures such as ASLR or NX. Or various operating systems.

Common refrain that things like OpenSSL counter nicely. Massive use, open code, and basically no code quality. Instead, all of them seem to have bugs and exploits available regularly throughout the year. You'd think using an obscure program would make things worse but software's baseline is so low it's often not the case. Instead, the high-quality stuff in both closed and open source seems to be made by skilled developers that care about quality. Only common denominator. Note, though, my recommendations are to use obfuscation on this with good, security controls in place plus well-coded apps. On to that.

"How do you decide that a piece of software is "well-written"? Even very strict review processes missed critical issues... and none of those are team for rare software."

For binaries, you can look at the software itself and how it runs to get a decent handle on how well it's written. Classic example is Bittorrent vs uTorrent. uTorrent had amazing amount of functions in tiny executable that loaded and ran fast without any issues for me. Clearly a great programmer wrote it. Adobe Reader vs Foxit has similar properties suggesting people at Foxit at least care about quality of their code and make a good attempt. You can also apply tech that isolates or hardens the excecutables themselves to care less about it. Nonetheless, had I chose Reader, the many eyeballs effect would've meant I'd have a new attack on my hands about every month (or more) whereas Foxit users weren't being affected as much. Tide might have shifted where it got popular enough & secretly had terrible enough code that CVE's are popping up all over the place. I haven't followed Foxit since I've been off Windows. I'd ditch it at that point if it got that bad. It will have done its job for a long time in reducing my headaches vs Adobe, though.

Now, for FOSS or even shared-source proprietary, you have even more options. The first is simply looking at the code to see if it's well-organized, has proper checks, has tests, and so on. For PDF readers, I remember Marc Espie telling me MuPDF had really, good code. Great coders like him know good code when they see it. More objective metrics might be found by running static analysis tools or something on a bunch of them to see how many warnings or issues show up. Lower the better. Also, remember my methods don't work in isolation: picked an obscure PDF reader, ensured it had noticeably better quality than most-targeted one, and with source may applied a transformation that added more checks and stuff.

"the proper advice is to reduce attack surface, not rely on some allegedly obscure piece of software."

The proper advice is to do both. Each technique I listed reduced issues either arguably or in the field. Testing it was easy: honeypots with known vulnerable stuff. Attacks didn't work. Obfuscation by itself had great benefit. So, combining strong security such as separation kernels, safer languages, static analysis, and so on with such obfuscation just makes strong security even stronger.

"Your homegrown CTR crypto might expose you for a related key attack. How do you know, since nobody targets it, until it is broken?"

Do you have references on this? Let's focus on the strong one to prove or disprove the principle: using one to three AES candidates with separate keys each in the way it's supposed to be used. My only modifications to AES candidate in isolation are using one after the other, randomizing the initial value of the counter, and changing the order of cipher application per session. So, it's AES-256/CTR + Twofish-256/CTR + Serpent-256/CTR in order or some other combo of them. Keys and counters come from a CRNG or TRNG. No repetition is allowed such as Twofish followed by Twofish in case theres internal interactions. Wheres the vulnerability in this construction that can be exploited by someone possessing only ciphertext?

"Guessing target architecture given an exploitable but is trivial. You need a simplest of data leaks."

It is but they rarely do. Field results show they just basically never do. It has to be a highly-determined, smart attacker that also knows odd ISA's. That's why I have other things waiting for them. Randomized instruction sets generated per application with enforcement doing with modified CPU's is the state of the art in this IIRC. I recommend stronger methods like in Hardbound or CHERI architecture that arguably immunize you to specific issues. Yet, CPU obfuscation often blocks riff raff and sometimes smart attackers constrained by time or boredom.

"There aren't many options available either."

That's true. Yet, how many Super-H coders do you know in malware market? Or even Alpha? Economics and laziness are why it still helps despite limited options.

"You already have to tangle with much stronger security measures such as ASLR or NX. Or various operating systems."

ASLR and various operating systems doing such things are in my list. We totally agree on them as nice speed-bumps for attackers. Also recommended some things that do full-safety or microkernel with small TCB + resource isolation. Even stronger. Personally, I'd probably add obfuscations like ASLR to them, too.