I am not yet "very old" (still not 40 :), but I've been called that by much younger folk, and I even joke about being "very old" in different social settings.

It's obviously a relative term, so I would hope this isn't considered "rude" more generally.

Even objectively, if life expectancy is 75 in a region, saying that someone 70+ is "old", or someone 85+ is "very old" sounds "correct" rather than rude. Is "very young" rude as well?

Is it rude to say that someone is very short or very tall? I mean, that 6ft basketball player is very short.

Yes, some will take offence at any objective statement of fact, and some of these are cultural (tall is good, short is not; young is good, short is not), but if your point is not to denigrate, then you are not rude imho.