How does a new engine find webpages at start? Does it work from the Dns system and indexes every domain name? At a certain point it will follow links I presume, but how does it start?
[Disclaimer: work at Cliqz]
We do not crawl the web in the traditional sense, our search was bootstrapped on query logs. It is the very reason we could succeed in building a search engine with minimal resources, in comparison to our competitors.We have written about this in a lot more detail here :
What about tools for power searchers ? Google have abandoned us.
Are you planning to create strong tools in that area ?
For example: custom search engines, the NEAR operator, limit search to sites that don't update that often or aren't linked to very strong sites(against SEO), etc
We speak about this is much more detail in this post (https://0x65.dev/blog/2019-12-05/a-new-search-engine.html), but in short, we prototyped our search initially with data we purchased from data-brokers. Once the concept was proven and HumanWeb was deployed (2015/2016), we rely only on our data.
The "last time Cliqz came up" can be found here: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu.... We have been posting multiple articles on our tech blog, explaining what we do and how we do it in great details. Your link points to an old thread of more than two years ago.
> We have been posting multiple articles on our tech blog, explaining what we do and how we do it in great details.
it's possible to have both great tech and loose morals - the two are not mutually exclusive, and one does not absolve the other (e.g. facebook's social experiments)
has there been a followup to any of the points brought up in the reddit thread?
There is plenty of documentation on data collected (see first posts regarding Human Web on the tech blog), how anonymization works, why record-linkability on data collected is prevented (and forbidden), etc. Furthermore, source code can be inspected, as well as traffic in the case documentation is not enough. I believe that is a better proxy to assess "morality" than random accusations on reddit or opinions formed solely on a half-baked press releases.
Do we need to refute all miss-conceptions and FUD that might arise due to the fact that 1) we collect data to build our services (search) and 2) we are funded by a media company (VCs seem to be more pure for an unknown reason).
The answer is no. Cannot recall who said that it takes much more effort to refute BS than to generate it. (That does not go for your comment in particular, that's why we replied, but for many of the comments and some of content of the subredit that you mention.)
> There is plenty of documentation on data collected (see first posts regarding Human Web on the tech blog), how anonymization works, why record-linkability on data collected is prevented (and forbidden), etc. Furthermore, source code can be inspected, as well as traffic in the case documentation is not enough.
Question is, is it opt-in data collection or do you make the choice for me? If it's opt-in, great. Otherwise, I don't want to read your "plenty of documentation" and so on and so forth.
If I remember correctly, Yahoo Open Source their current and next generation Search Engine Vespa [1], why wasn't that used, and instead starting from scratch?
I like how the blog post title is, likely, based on Google's Page and Brin seminal paper "The Anatomy of a Large-Scale Hypertextual Web Search Engine". :)
This blog post, and other in the series, mention RocksDB is used for the index, but it's not explicitly described how and to what end. I 'd love to know the details.
Thank you for great article. I've been writing my own search engine during last 3 years and it's funny how similar my setup is to yours with K8S/Kafka/Streams/Go/RocksDB.
Actually about RocksDB - are you using it from Go(via gorocksdb?). Now and then I have hard time optimising RocksDB and still have very loose understanding how much RAM it will consume
[1] https://0x65.dev/blog/2019-12-01/the-world-needs-cliqz-the-w...
[2] https://en.wikipedia.org/wiki/Cliqz
[3] https://en.wikipedia.org/wiki/Hubert_Burda_Media