Most titles use one of the standard Nintendo-provided microcodes for the RSP (Reality Signal Processor, the programmable part of the graphics chip).
F3DEX2 ("Fast 3D, Extended, Version 2") is one of the well-documented ones and one of the ones used by most games. You can find a breakdown of the command stream here: https://wiki.cloudmodding.com/oot/F3DZEX
But I'd bet most titles had their own formats to support vertex deformation and animation. (Anybody know any details?)
It looks like it'd be pretty fun to program the N64 GPU, with its "high-quality, Silicon Graphics style pixels" :) https://level42.ca/projects/ultra64/Documentation/man/pro-ma...