in a couple of projects I worked on, we also stored visited urls in a set of bloom filters, also stored in flat files on disk.

At some point querying the db to check what URLs you have can become quite heavy