you're right. also, there's a type of responsibility i'm hinting at here that's less formal, more like: "this person can handle tasks of complexity X and is willing to do what it takes to create a quality solution", that is the nuts and bolts to my mind of a successful and high functioning team.
That’s what happened to me. I left a job as the dev lead at a medium size company with a small development department to work at a small software company where so specifically negotiated not to be a team lead. I told my now manager that I could be a lot more effective working across teams as needed as an IC and just demonstrating and mentoring the team leads who I felt were much more qualified than I was to be a team lead. They knew the software stack and they were all much stronger on the front end than so was. They all needed help with infrastructure and process.
I make more now and have more influence using relationship and expert power than I ever did with the role power I had before - the three levers of power in an organization.