- Laser off the part marking. Not knowing what a part is makes the job much more difficult
- One time programmable chips: can't modify or read off firmware if the JTAG bus is disabled
- Encrypted firmware: helps if someone is able to fuzz the chip to dump the firmware
- BGA parts: hide the pins, bury the traces. It makes the job harder but not impossible
- Programming before soldering: you can leave the programming pins disconnected so someone would have to remove the chip before attempting anything on it
- Use more advanced features of the chip: some chips offer secure memory locations that can contain decryption keys, magic numbers, whatever you want. You could have a magic number that you XOR with every literal. It would certainly make things more difficult to determine what is what in the assembly code if you could decrypt it
- Pour some epoxy over the chip or board: makes repairs impossible but also can screw over the reverse engineer.
- Work with a manufacturer to build a custom chip. You could do crazy things like move the programming pins around and hide them as other things. Like the JTAG test points would be random decoupling caps hidden in the board.
- Finally, threaten to sue anyone that publishes anything
You can't. It goes against the very nature of the medium, like trying to delete something on the internet. If it's something a CPU has to execute, it has to be in memory where it can be dumped. At best all one can do is make it harder to stop less determined adversaries.
That said, there actually is one nasty [1] workaround: run some critical functionality on a custom USB dongle that the user has to have connected in order to use the software. It could be a calculation in a critical path that's not compute bound but without which the software is unusable. It could even be a JIT engine that consumes encrypted code and returns polymorphic executable code designed to be near impossible to assemble back into a static binary. Some fabs can make tamper-resistant ASICs with a specialized packaging process that couples the on chip memory to the package so that opening the package makes the memory unrecoverable for extra security. This level of protection would be effective against all but the most determined and well funded nation state or competitor.
[1] Nasty for the user, the developer, and the investor all in one!
That's basically what the casino gaming industry is. They're done of the most physically secure systems against reverse engineering that you're likely to run across. More so than atms or a lot of other secure systems.
- Laser off the part marking. Not knowing what a part is makes the job much more difficult
- One time programmable chips: can't modify or read off firmware if the JTAG bus is disabled
- Encrypted firmware: helps if someone is able to fuzz the chip to dump the firmware
- BGA parts: hide the pins, bury the traces. It makes the job harder but not impossible
- Programming before soldering: you can leave the programming pins disconnected so someone would have to remove the chip before attempting anything on it
- Use more advanced features of the chip: some chips offer secure memory locations that can contain decryption keys, magic numbers, whatever you want. You could have a magic number that you XOR with every literal. It would certainly make things more difficult to determine what is what in the assembly code if you could decrypt it
- Pour some epoxy over the chip or board: makes repairs impossible but also can screw over the reverse engineer.
- Work with a manufacturer to build a custom chip. You could do crazy things like move the programming pins around and hide them as other things. Like the JTAG test points would be random decoupling caps hidden in the board.
- Finally, threaten to sue anyone that publishes anything