Then those authors are idiots. Libraries pay for their books, they don't seize them for free through some kind of literary eminent domain.
If their argument is that it should be illegal for anyone to share a book that they've paid for with someone else, that would be a radical redefinition of what it means to "own" a book.
My mom borrowing my book is qualitatively different than a public institution set up for the mass renting of media. The law makes that distinction all the time; for example, the CRA prohibits discrimination in a restaurant, but not at my potluck.
Back during the 90s, video rental stores would have to pay a higher price (around $100 IIRC) for a copy of a movie they were licensed to rent. If you believe that media should be licensable, then it follows naturally that more permissive licenses should cost more.
The main reason that libraries are morally acceptable, by the way, is that they are viewed as institutions of learning and betterment that service students, the unemployed, and the less fortunate. If libraries were focused on entertainment instead, they would not be so highly regarded and their copyright issues would come to the fore.
If their argument is that it should be illegal for anyone to share a book that they've paid for with someone else, that would be a radical redefinition of what it means to "own" a book.