I've been meaning to attempt running a custom search engine for particular sites I've 'bookmarked'. Some sites contain gold that could be useful in the future and is not often discovered in Google results.
Should I go the Postgres/Elasticsearch route or are somewhat out-of-the-box solutions available?
For such a light demand and fixed site requirements, a single-file sqlite dB is probably best. Modern Sqlite has full-text capabilities that are quite powerful and relatively easy to implement.
It depends on what the user requirements are. FTS works pretty well with both Postgres and SQLite, in my experience.
Here's a git repo someone can modify to do a cross comparison on a specific dataset, if they are interested. It doesn't seem to indicate the RMDBs are outclassed in a small-scale FTS implementation.
The article covers typo resilience in the section "Typo tolerance / fuzzy search".
This adds a step between query entry and text search where you find the similarity of query words to unique lexemes if the word is not a lexeme. Seems like a reasonable compromise to me?
I'm not trying to be argumentative. As long as people find a solution they're happy with, I think that's great. For me, I'm far less interested in handling typos, but I can see how it would be valuable in many applications. I'm usually less interested in tying in and learning another set of services if I can get 90% of the way there with one, but leaving the option of adding it later if additional requirements make it necessary.
I am wanting to do something similar. Archivebox seems to be the best solution for this sort of self-hosted, searchable web archive. It has multiple search back-ends and plugins to sync browser bookmarks (or even history).
I haven't finished getting it set up though, so take this recommendation with a hefty grain of salt.
ArchiveBox can extract text from HTML (and possibly PDFs too). I think it can be configured to extract subtitles from YouTube videos as well. So it can do full text searches. Basically you could have your own, offline & curated search-engine.
You could run a full text search or search against an auto-generated summary. Or if you want to be fancy, use semantic search like in Retrieval Augmented Generation.
Edit: I forgot to add how would I add the webpage to the databases already suggested here? Do I need to use a separate program to spider/index each site, and check for its updates?
If you're looking for a turn-key solution, I'd have to dig a little. I generally write a scraper in python that dumps into a database or flat file (depending on number of records I'm hunting).
Scraping is a separate subject, but once you write one you can generally reuse relevant portions for many others. If you can get adept at a scraping framework like Scrapy you can do it fairly quickly, but there aren't many tools that work out of the box for every site you'll encounter.
Once you've written the spider, it's generally able to be rerun for updates unless the site code is dramatically altered. It really comes down to how brittle the spider is coded (i.e. hunting for specific heading sizes or fonts or something) instead of grabbing the underlying JSON/XHR that doesn't usually change frequently.
Depending upon the type of content, one might want to look into using the Readability (Browder's reader view) to parse the webpage. It will give you all the useful info without the junk. Then you can put it in the DB as needed.
Having never seen it, I have no context for which details were left out or bent to fit the narrative, but this still feels amazing. Well-earned sunglasses
Moving abroad and making some sort of life for yourself is two things: 1. a review of your preconceptions and a test for your ability to adapt, 2. if you stick it out, you may integrate new ways and viewpoints into who you are and what you think.
When you then go back to people who are similar (because that's always going to be the case to some extent, even if locally the community feels and/or thinks itself diverse) you will frequently feel that others have not 'progressed' or are 'stuck' in those culturally determined ways. It's hard to share that growth with people that haven't done the same, which is I think why expats stick together, even when they're back.
I moved between Western countries I should perhaps add.
I’m not the person you replied to, but for many of us “home” is not a first world or Western country. The opportunities there are much fewer, quality of life is lower in many tangible ways - do you like having reliable power, internet, mail/shipping, safe roads? Dangerous crime is a much more serious problem, and corruption is rife. Healthcare is subpar if you’re not wealthy. Education is generally poor, which has all sorts of consequences for society as a whole.
There’s also the fact that after living elsewhere for many years, the home you go back to is not the same as the home you left.
I've been wanting to run my own search engine sorta thingy that indexes websites I feed it. I sometimes find little nooks of the net that post resources I may need in the future. Like my own mini-Google that indexes a list of sites.
How can I go about creating this? Are their off-the-shelf solutions, will I need to say combine scrapy with elastic search? The links in this thread look promising.
Should I go the Postgres/Elasticsearch route or are somewhat out-of-the-box solutions available?