I was founding engineer ("CTO") of a startup a couple years ago, and I completely agree. We had the CEO who had domain expertise in our market, but who had never ran a tech company and was not technically proficient. I on the other hand was technically proficient, but I also never ran a tech company.
This lack of execution expertise doomed us; we really needed a strong operations role. We made good progress on the product itself, but hit a wall when it came time to ask for money from investors.
I was in a similar position, working as CTO under a first time CEO where we built a small team, product, got some limited traction, but follow-on execution completely failed. It was incredibly hard. It depresses me to this day, thinking about where it all went wrong. I probably won't ever do it again.
This lack of execution expertise doomed us; we really needed a strong operations role. We made good progress on the product itself, but hit a wall when it came time to ask for money from investors.