I think we imagine the models have emergent properties, because we have never in human history been able to have a meaningful two way conversation with non human entities before. So we believe in them. Remember how in the 1970s people believed that computers must be correct. There was no understanding of how they actually worked.