Well .. they do flow in the conductor, and in particular they flow through silicon junctions or none of the semiconductor physics would make sense, but the energy is carried in the field which is indeed outside the conductor.