With vectorization and compiler optimizations* I doubt pure Python, well CPython running pure Python, would match Fortran and some C open source solvers that are the state of art like Lapack for dense matrices and superlu or plastix for sparse ones. It's not much because of the language, but mainly because python has more overhead than C or Fortran.
Maybe as you said python can work well in solving a single linear system using GPU. I worked with this before GPUs came to be used for that, so I can't comment on this one.
* Generally compilers optimize mathematical expressions unless they involve complicated pointer arithmetic, this is why Fortran is still used in this area, Fortran does not have this feature.
In numeric Python packages computations are never done inside the CPython runtime, they're done in specialized kernels written in C, Fortran and sometimes fine-tuned assembly including a sampling of the various BLAS/LAPACK flavors... so yes NumPy can often be competitive with or outperform libraries written in lower-level languages.
With vectorization and compiler optimizations* I doubt pure Python, well CPython running pure Python, would match Fortran and some C open source solvers that are the state of art like Lapack for dense matrices and superlu or plastix for sparse ones. It's not much because of the language, but mainly because python has more overhead than C or Fortran.
Maybe as you said python can work well in solving a single linear system using GPU. I worked with this before GPUs came to be used for that, so I can't comment on this one.
* Generally compilers optimize mathematical expressions unless they involve complicated pointer arithmetic, this is why Fortran is still used in this area, Fortran does not have this feature.