I used AFL (fast) on the TXR language and fixed a few issues as a result in release 148. None of them were corruption issues. Several were degenerate behavior. One was a termination via abort, due to something not being checked. I just fuzzed the parser, not even delving into semantics very much. Still, the parser involves expansion of macros and special operators, and is complicated enough in itself, so things can go wrong.
One issue that was uncovered was that the defvarl macro was improperly implemented. It is supposed to be like defvar, but not mark the symbol as special, giving rise to a global lexical variable (hence the "l"). However, it was actually behaving as a synonym for defvar. How AFL helped uncover this is that its fuzzing generated the form (defvarl) with no parameters. Now defvarl was marking the symbol special at expansion time. There was no error checking for the missing name, and so the symbol nil was passed to an internal function to mark a symbol special; and that asserted on nil being a non-bindable symbol. Of course, my reaction was, huh? How do we end up in mark_special when defvarl is being expanded? Oh shit ...
Another thing AFL found was degenerate behavior in the op operator for generating anonymous functions. op scans its body for numeric parameters, and then generates a lambda with a number of arguments based on the highest numeric parameter: (op whatever @1 @2) gives you, ignoring certain other details, (lambda (arg1 arg2) (whatever arg1 arg2)). AFL figured out that there is a problem if we do something like (op whatever @1111111111111111111111111111111). The expander wants to create lambda syntax with a huge number of parameters, which blows up memory. Though nobody would do this in a normal program, suppose that you have some application which accepts code and interprets it. Of course, you sandbox all the stuff which could let the code access the system---but issues like this means that you're open to DoS before the code even runs, just from parsing it and expanding its macros. You wouldn't sandbox off a useful programming feature like op. I just put in a hard maximum on the integer that can be used in the op syntax.
One more degenerate behavior AFL uncovered was in backquote nesting. The TXR Lisp backquote is spelled ^ (caret). AFL uncovered that ^^^^^^...FORM nesting blows up memory exponentially in the number of rounds, due to each round of nested backquote expansion multiplying the size of the expansion. This prompted me to finally add some optimization of the generated code to the sloppy backquote expander, which reduces the expansion growth to linear in the number of rounds.
After these fixes, I noticed speed increase in "make tests" in excess of 7%, too.
Honestly, it's not really surprising. Emulators, and, more broadly, video games in general, tend to not give little to any thought about security.
To be fair, exploiting games and emulators directly aren't a particularly common attack vector. I've never heard of any wild malware that attempted to exec itself via a (legit) emulator, although IIRC arbitrary code exec on clients had popped up a few times in the wild on older multiplayer games (mostly CoD and various Source multiplayer games). Most of the "malware" stuff related to emulators I've heard of are cheap tricks (e.g. tainted emulator binaries on shady websites, EXE files deceptively labeled as ROMs) and not any fancy exploits.
Actually, video games are the main exploit vector for game consoles. Out of my memory there was the name buffer overflow in Zelda for Wii, and a font corruption issue in King Kong for the Xbox 360, which both were used to allow running homebrew software.
Attacking video games for code execution was the main entry point as the devices transitioned into semi-connected devices (the original Xbox, Playstation 2, Gamecube and then later the Wii), that has mainly transitioned now to "attack crusty old versions of Webkit!" for the current batch of consoles
Attacking emulators through games for code execution however is fairly novel
I wonder why emulators aren't a common attack vector. They're fairly popular, and downloading ROMs from mysterious shady sites is par for the course with them. If you could subvert a popular ROM site, or even just SEO your way to the top of the Google listings, you could get a lot of people.
Probably because they'd just infect the emulator with some payload rather than figure out an exploit vector that depended on the payload being opened in the emulator later.
I'd prefer if these emulators would just run in a sandbox so an exploit could affect your save files and nothing more. They are so complicated there will always be bugs.
One issue that was uncovered was that the defvarl macro was improperly implemented. It is supposed to be like defvar, but not mark the symbol as special, giving rise to a global lexical variable (hence the "l"). However, it was actually behaving as a synonym for defvar. How AFL helped uncover this is that its fuzzing generated the form (defvarl) with no parameters. Now defvarl was marking the symbol special at expansion time. There was no error checking for the missing name, and so the symbol nil was passed to an internal function to mark a symbol special; and that asserted on nil being a non-bindable symbol. Of course, my reaction was, huh? How do we end up in mark_special when defvarl is being expanded? Oh shit ...
Another thing AFL found was degenerate behavior in the op operator for generating anonymous functions. op scans its body for numeric parameters, and then generates a lambda with a number of arguments based on the highest numeric parameter: (op whatever @1 @2) gives you, ignoring certain other details, (lambda (arg1 arg2) (whatever arg1 arg2)). AFL figured out that there is a problem if we do something like (op whatever @1111111111111111111111111111111). The expander wants to create lambda syntax with a huge number of parameters, which blows up memory. Though nobody would do this in a normal program, suppose that you have some application which accepts code and interprets it. Of course, you sandbox all the stuff which could let the code access the system---but issues like this means that you're open to DoS before the code even runs, just from parsing it and expanding its macros. You wouldn't sandbox off a useful programming feature like op. I just put in a hard maximum on the integer that can be used in the op syntax.
One more degenerate behavior AFL uncovered was in backquote nesting. The TXR Lisp backquote is spelled ^ (caret). AFL uncovered that ^^^^^^...FORM nesting blows up memory exponentially in the number of rounds, due to each round of nested backquote expansion multiplying the size of the expansion. This prompted me to finally add some optimization of the generated code to the sloppy backquote expander, which reduces the expansion growth to linear in the number of rounds.
After these fixes, I noticed speed increase in "make tests" in excess of 7%, too.
I look forward to more fuzzing.