The only downside would be time.Time structures grow by 8 bytes, but this is really minor because if you're storing times you're likely storing timestamps and not time.Time structures.
Actually, it only grows with 4 bytes on the 32 bit systems (no growth on the 64 bit). It's right there in the doc:
>On 64-bit systems, there is a 32-bit padding gap between nsec and loc in the current representation, which the new representation fills, keeping the overall struct size at 24 bytes. On 32-bit systems, there is no such gap, and the overall struct size grows from 16 to 20 bytes.
