Exactly: 99% of the time, "webpack is slow" is just code for "I have Babel in my toolchain" and forgetting to set ts-loader as "transpile only".
Personally I removed it years ago, and Webpack 5 even allowed me to get rid of more loaders now that there are Assets Modules that automatically detect assets and webworkers using the "new URL()" syntax, and Typescript does everything else I need.
Even if your target platform didn't have ES6 modules, Babel is probably not the tool you'd want to transpile those with. Let the bundler (which is the topic of the discussion here so I assume a bundler is used) handle them. Tree shaking works better that way anyway.
Also if JSX or similar JS extensions are the only thing you need tramspiling for, you might want to look at Sucrase [1] as a fast alternative to Babel.
Personally I removed it years ago, and Webpack 5 even allowed me to get rid of more loaders now that there are Assets Modules that automatically detect assets and webworkers using the "new URL()" syntax, and Typescript does everything else I need.