I presume when you say "remove from the search results", you mean removed from the first page or top n spots or whatever.

But what if there are 20 good resources? They can't all be on the first page, or in the top spot.

Perhaps yhe other sites improved, or were better in some way?

Being on top for a while is no guarantee you'll be on top forever. It's precisely Google's business to change the algorithm.