Increasing the size of the court isn’t a slippery slope to somehow making justices elected by states/districts. It’s not like as soon as you get too many justices it turns into a legislature.

The interesting differences between the legislative and judicial branch is not the number of people (moreover, the Supreme Court is not exactly the entirety of the federal judicial branch).