It's hidden weirdly.... go to "Keyboard" prefs pane and "Keyboard" tab (initial one shown when you bring it up.
At "Touch Bar shows".... change that from "App Controls with Control Strip" to "Expanded Control Strip". Then you will see the individual buttons. You can then further "Customize Control Strip..." and you will now see more buttons. But you will also loose "App Control".
Of course, why can't those specific buttons be a customization option there with "App Control with Control Strip" forcing you to choose between volume up/down and "App Control"? Because Apple.
Savior! Thanks. I feel really dumb for not knowing about this, but my ego is going to blame Apple. Just in general feels rushed all over for many many things - this is (was) just my personal biggest gripe!