The point isn't that everything is a file; it's that you can treat everything like a file.
Of course the abstraction maps varying degrees of well, but it also simplifies the hell out of the lion's share of the ways we interact with computers, and serves as a foundation for the rest.
Than what, an entry in some table in the kernel that you can use to read bytes from the thing it represents and/or write bytes to it? That's what a "file" is in *nix.
The on-disk notion of a file sits atop layers of that abstraction. It doesn't, for example, go away when you `rm` it. An entry in a table does, but if some other process had that (disk) file open at the time, its data is still there, readable — and recoverable — through the former file's entry under that pid's /proc/$pid/fd/.
