You might try the Audio MIDI Setup utility. This allows you to add/remove devices which might work to remove the microphone as an output device.

If you're looking for a fix and more advanced customization you could try Soundsource: https://rogueamoeba.com/soundsource/