I reject the virtual environments and have no issues. On an untrusted machine (see e.g. the recent token leak):

  /a/bin/python3 -m pip install foo
  /b/bin/python3 -m pip install bar

The whole venv thing is overblown but a fertile source for blogs and discussions. If C-extensions link to installed libraries in site-packages, of course they should use RPATH.