You might want to check this out then: https://haim.dev/posts/2020-07-28-dual-monitor-kvm/

That was on HN earlier this year. Basically it's software that you run and set up to detect when certain peripherals connect/disconnect and have it switch the input on the monitor automatically. I've been using it and, while not always perfect, it actually works pretty well, especially considering the price.

If you don't always want the monitor to switch inputs (sometimes I just want to quickly switch my keyboard/mouse to my laptop to do something but don't want to switch monitor inputs) you can also just set up shortcuts or scripts (I created an Alfred workflow, for example) and use something like https://github.com/kfix/ddcctl to switch monitor inputs.

You can buy cheap Displayport switches that work just like the usb one you have.

That’s the plan once I upgrade. For now I’m still using my 2015 mbp, so it’s displayport from my desktop and HDMI from the laptop.

Finding a cheap switch capable of doing both is not that easy