tangential question: if i wanted to poke around with the risc-v isa (like add an instruction) and then wanted to add support to an emulator, compiler and operating system for that instruction, is there an emulator, compiler and operating system that are best suited for this sort of isa experimentation?
The CHERI team had the same problem: https://github.com/CTSRD-CHERI/cheribuild. You'll need something along the lines of "./cheribuild.py sdk-riscv64 freebsd-riscv64 disk-image-riscv64 run-riscv64", bit of disk space, and a free couple of hours for it to finish building everything; at the end you'll get a login prompt for the guest.
QEMU TCG is not that difficult to understand, and has quite comprehensive support for RV64 already, so you could add a new instruction there. As for compiler/kernel support, that's probably a lot more difficult, but it's hard to say what would be involved (eg. deep changes, or simple __asm__ statements) without knowing what kind of feature you'd be trying to add.
what if i said risc-v isn't as important (to start), but rather just a dead simple isa with a dead simple emulator, compiler and operating system (like something that can take cross-compiled binaries and run them with basic input and output).
like i'd like to experiment with novel ideas in the area, but ideally would like to avoid getting into the complexity of real compilers, operating systems and emulators until the ideas are looking proven...
i can go looking more for stuff myself (maybe toy mips stuff used in schools or something) but this seems like something that people would do and therefore there'd be known tooling?
I built a RISC-V emulator in python focused on extensibility and debugability for this exact purpose. It should be dead simple to use with plain RISC-V Assembly files. It has a userspace-only mode and a minimal privileged mode: https://github.com/AntonLydike/riscemu
Adding new instructions is as simple as adding a method to a class, as long as you are reading straight assembly files. If you want to parse ELF-files, you also have to add the opcodes for your command to the decoder (which is currently not very well documented). If you'd like to do that, feel free to open an issue and I'll work you through it.
I also built a minimal kernel, which works on the emulator, but it doesn't support IO and is very rough around the edges in general.
If you're asking "do simple teaching OSes/toolchains exist?", of course they do (Minix is the most famous one). If you want risc-v specifically, with a quick search I found xv6, which seems to have been put together for https://pdos.csail.mit.edu/6.S081/2021/ .
Another alternative is to look at something like PicoRV32. There's no "OS" as such that can run on PicoRV32, there is only a few kb of RAM, so single C files that you write from scratch. But it's a simple project and fun for hacking. You can put it on a cheap Lattice FPGA using a completely open source toolchain. A fun way to get into hardware / FPGAs for sure.
If you're into the open ISA idea but find the big guys a bit intimidating, you might have fun with OpenRISC. At least lately I've had a blast hacking on it. The kernel and QEMU implementations are very simple, and Stafford Horne is fun to talk with.
If you just want to test the idea without any major complexity, like "I just want to add an instruction for int8 dot products" or whatever, you can write your own emulator for RV32IMC, in M-mode, in a few hundred lines of most modern programming languages. There are a bunch of test suites to check the base ISA works, and you can use a C compiler to target bare metal, and use GCC's .insn directive to embed custom opcodes inline. Run a bunch of tests this way. It's actually very easy to control the whole stack in cases like this.
Once you get into actually needing to boot a system like Linux, you're going to need more support from the emulator, and at that level you'd either want to look into extending the Spike emulator, or QEMU itself. Like, features that interact with different privilege levels or whatever are just complex, there isn't any way around just working on the source directly. I realize that's not a very good answer.
There are simpler kernels and whatnot you could also probably use. I mean, I would hope so. Honestly, a lot of it depends on the specifics of what you're looking at trying to implement. Simple instructions that work in M/U mode are the best place to start.
I suggest trying to write your own emulator first. It'd surprise you at how easy the base RISC-V IMC ISA is, and you can add as many custom instructions as you like, and you'll have a C compiler (and Rust too, actually) to start with.
Emulator is dead easy other than bit twiddling initially.
The compiler aspect would depend because although compilers try to construct the code generator automatically (e.g. by matching on a tree) sometimes it requires a special case to do good code.
