Not at all. If you just reference to a package by its name, that is good enough for a compiler. I think automatic download of packages from third party sites doesn't belong into a language and its native tooling. The downloading and installing of third party packages could be done by a tool provided together with the language core tools, but should not be required to be able to compile a project at all.
Strictly speaking, as with cargo/rustc, the Go module system and Go compiler are separate, and the latter can run without calling the former. There are various flags to tell the "go" command (which is a frontend to other tools) how to behave when modules are involved, e.g. so that it works well on airgapped networks and locked-down intranets or can be used safely with untrusted source. You can also still vendor modules to ensure they are locally available.
