Rust doesn't have a GC in the first place, so there's nothing to turn on. Even then, the fundamental hurdle with GC isn't the effect on your program's runtime, it's the effect that it has on the lifetimes of your data. GC (and RC) are means of dynamic lifetime determination. Manual memory management is static lifetime determination. The latter requires you to structure your code in a specific way, which may be less convenient for the programmer depending on the application. Converting dynamic lifetime determination to static lifetime determination then requires changing the very structure of the program itself (including how APIs work) while preserving semantics, which is beyond any existing tools.

I was just thinking that most languages are either GCed (like Java or Go) or manually curated (C/C++/rust).

My idea is that if one could write in a language like Go (which has some kind of GC), but when one wants, say "I'll take care of this memory".

Rust actually started with a model like this. The idea was when starting out for the first time you can use garbage collected pointers for everything and then switch to the manual model when you needed it. Problem is then that unless your standard library and all the major open source libraries work perfectly for both the garbage collected objects and for the manually managed objects then you need to learn manual memory management early on anyway, so the garbage collector wasn't helping anyone. The language design element of this is really tough to do in a usable way.

D has optional GC but their standard library is kind of fucked up.

Yes, my above paragraph doesn't imply the impossibility of intentionally mixing manual and automatic memory management. E.g. you can kinda do this in Go, but the language provides no facilities to help you get the manual bits right. And you can kinda opt-in to GC in C++ and Rust (via reference counting), but it's not super ergonomic.