It would make a lot of difference. First of all, you'd be able to run programs written in dynamic JVM languages like Clojure, Groovy or JRuby unmodified.

Second, Dalvik was designed for low-power, low-performane, previous-generation handsets. Embedded HotSpot is years ahead of it in terms of performane.

I honestly don't know if Android is too far along to switch.