So what's the point of the endowment then? To make the green numbers go up?

To be invested and generate returns which can fund the university's programs. As I said in another comment, it's a long-term investment, not a spending fund. You don't eat your seed corn.