This is mostly just raw data, it isn't that useful (yet).

The security issues with using curl directly to my terminal feel a bit dangerous. I'd rather use my browser and be able to see the results over a json tree. Providing raw access to ES has a high risk to reward.

The search results for https://dontbeevil.rip/search?q=python%20context%20manager%2... are non-topical hits on SO records. I even put the name of the python package (from stdlib no less) in the query string.

I was able to find what I needed on devdocs.io in less than 10 seconds.

https://devdocs.io/python~3.10/library/contextlib#contextlib...

In no way am I trying to discourage you, but until the basics are in place, search over arxiv abstracts is way less useful than just SO and docs (language and libraries).

I would recommend returning text/plain by default and .json if someone asks for it (in the url), no everyone can set headers. I'd also put an about page at the root of your site, plain text is fine.

> Providing raw access to ES has a high risk to reward.

I'm not quite that foolhardy :). The only thing that gets passed to ES is a sanitized simple_query_string.query. Should be relatively safe. I'm sure someone will prove me wrong though.

And you can't trust the server here because its publishing 3rd party content in the way of search results.

The risk reward ratio of mentioned above is to far in the risk for my liking

What?

I think it's fine to be paranoid here, the attack surface is massive.

Having thought quite a bit about the search space, I think a whitelist approach is going to be the next big search thing, because advertising and bs sites have corrupted SEO too far.

I'm reminded of the site indexer websites in the early days of the internet. Curation if done properly (based on quality of content and not certain other factors that currently play too heavy a role in the seo algo black boxes) seems to be how we adapt to the current information tsunami we are all dealing with.

I think a long time ago I decided I would even pay for such a service, just like I am willing to pay for a good new source (FT for me, not cheap, but worth it). Im not positive the 10$ mark is low enough but I hope for the general landscape it is.

Just dont forget to keep dontbeevil more than a catchphrase. In particlar, please be transparent with what user data you collect and how you use it.

It almost as to be. Spammers, growth hackers, et al. are just too numerous and too good.

> Im not positive the 10$ mark is low enough but I hope for the general landscape it is.

I saw enough people mention $10 that I decided to go for it. To be honest, $10 is already probably too low to be sustainable because of the huge amount of resources search consumes and the high cost of development.

My gut feeling is that it's economically impossible to build a good search engine that isn't loaded with ads and spyware. But I spent so long complaining about G that I decided to prove to myself one way or another.

I’ve been intermittently working on much the same idea as the OP, and I suspect this is actually a lot less of a problem than it seems, since they’re focusing on a niche. Indexing everything the way Google does requires a lot of resources, but indexing the majority of useful material in a specific domain takes a lot less. (My ElasticSearch index for the entirety of StackOverflow is a mere 40 GB, for example.)

By far the more expensive part is likely to be paying market rates for a developer (you need a decent number of users paying $10/mo to hit a mid-market salary), but in theory this scales relatively independently of userbase.

Edit: I’ve just noticed I’m replying to the OP, who’s mentioned downthread that they’re using BigQuery and spending $200/week. I’ve gone the marginalia.nu route and run everything on a computer in my living room, which changes the calculus somewhat—it’s a lot cheaper, but probably involves more development time.

For me it’s mainly about the learning experience but I’d be interested to hear your thoughts on the tradeoff.

Even 'javascript reverse string', which I expected some docs or stack overflow pages seems to give me a HN thread, someone's github repo and a not very related SO thread.

Is MDN, MDSN, more dev docs documentation on the roadmap?

It's definitely an interesting technique. Do you have anything in place to detect garbage, substenceless articles like which has started popping up on Google?

I've seen the occasional one using github repositories or pages. Looking at the current list you're broadly reliant on moderators and communities, and as the search engine you moderate which sites are indexed.

Indeed, it's explicitly for programming only (for now).

> Even 'javascript reverse string', which I expected some docs or stack overflow pages seems to give me a HN thread,

Next up is indexing language documentation. At this point I'm relying heavily on Q&A and community sites since they have their own built in quality rankings.

> Is MDN, MDSN, more dev docs documentation on the roadmap?

Most definitely. Feel free to dump a list of urls of your favorite doc sites. I'm building a whitelist now.

> Do you have anything in place to detect garbage, substenceless articles like which has started popping up on Google?

My strategy is to not index spam in the first place. That's why I started by extracting links from with community sites that have their own moderation in place. The next step is to whitelist high quality sites. That is potentially a huge list to maintain, which is why I'm am narrowly focused on software development.

Everything old is new again...

How are you using the "built in quality rankings", could you give some examples?

On Reddit, say, except in a few groups like AskHistorians you can still get very high ranking for a meme post and very low ranking for a list that has high informational value. StackOverflow is extraordinarily prone to killing off reasonably good contributions and giving very high ranking to out of date answers (the latter is the biggest problem with SO sites at present IMO).

Many of these are linked to GitHub/GitLab repos, so I'm not sure how you'll deduplicate that.

Here's how I am going to evaluate your search engine. Yesterday I searched Google for "get dynamodb table row count" and found this URL, https://bobbyhadz.com/blog/aws-dynamodb-count-items, which provides a terrible recommendation involving a full table scan.

With DontBeEvil, I didn't find the correct answer, to use the describe-table API.

If you really plan to dedicate a year to this, I would strongly encourage you to re-post again as soon as you have a strong update. Right now this has potential to provide value but really does not. So update us when you have confidence that you might be providing value! But we think you're on to a great opportunity.

They are, but there are relatively a few of them because my only page content source is the Common Crawl. The hit rate vs the total urls I'm interested in is not great. I expect to fix this soon.

I'm also not indexing entire sites, only specific upvoted urls. This will change as well.

> Have you considered using social user karma (this could be a 1-10 score uniquely calculated for users of each of HN, Twitter, Reddit as long as it's built in a modular way) as a weight in a PageRank style schema?

Definitely. I've already started in on calculating a rank coefficient for submitters, but it's not completely clear now to best use it yet.

> Here's how I am going to evaluate your search engine

Feel free to dump more of these. Some solid test cases would be very helpful.

  function rip {
    param (
      [Parameter(Mandatory, ValueFromRemainingArguments)]
      [String]
      $Query
    )
    $RequestParameters = @{
      URI = "https://dontbeevil.rip/search?q=$Query"
      Headers = @{ Accept = "text/plain" }
    }
    $Request = Invoke-RestMethod @RequestParameters
    return $Request
  }

  > rip heartbleed bug

  Heartbleed Bug
  <https://heartbleed.com/>    
  Heartbleed Bug The Heartbleed Bug The Heartbleed Bug is a 
  serious vulnerability in the popular OpenSSL cryptographic 
  software library....

Edit: obviously you can query it from a browser, but it would take like a couple hours to have a view that parsed the json and put it in a google-style layout with a search bar.

It's coming. I decided to get it out there as soon as I had an index that could theoretically be useful. The feedback I'm getting will drive the next chunk of work that gets done. For instance, I'll probably bring in language docs next as a lot of people have asked for them already.

I think it's a good bet. Would expect to be a LOT harder for a "growth hacker" to make his way up in HN points as opposed to Google rank.

    >alias rip="curl -G -H 'Accept: text/plain' \
    --url https://dontbeevil.rip/search --data-urlencode q="

    >rip hello
    {"message": "Missing required request parameters: [q]"}
    curl: (6) Could not resolve host: hello

    rip() {
        curl -G -H 'Accept: text/plain' \
        --url https://dontbeevil.rip/search --data-urlencode "q=$@"
    }

(I'm acting all wise, but I learned that today as well :) )

  #! /bin/sh
  query_string="q=$@"
  curl --get \
    --header 'Accept: text/plain' \
    --url https://dontbeevil.rip/search \
    --data-urlencode "${query_string}"

   #!/data/data/com.termux/files/use/bin/env sh
  curl -G -H "Accept: text/plain" \
  --url "https://dontbeevil.rip/search" \
  --data-urlenconde "q=$*"

Edit: typo in your version: "urlenconde"

Autocorrect does make me laugh some days :D

  function rip() {
    printf ">>"
    read query
    curl -G -H 'Accept: text/plain' --url 
  https://dontbeevil.rip/search --data-urlencode q="$query"
  }

Is it just /r/programmer? or many other programming related subreddits?

In general though, this is great. I would similarly love a solution where we could submit sites to be indexed. A way to have a search engine for all the websites I want specifically would be awesome. You could probably add some sort of popular filter on top of it so that only sites popular enough get indexed. I don't know. Just an idea.

I love the fact that it's accessible from the terminal. That's fantastic. Although, would be nice to have a very simple HTML front-end. Think very early Google or go very brutalistic.

Anyway, excited to hear about it.

Edit:

Doing the following gives me an Internal Server Error for some reason.

curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode 'q=Notes' {"message": "Internal server error"}

There's about 30 programming focused subreddits.

> I would similarly love a solution where we could submit sites to be indexed.

This is on the plan. I want to allow common interest groups to maintain their own search verticals. I also want to allow individual users to add everything from bookmarks to notes (privately only of course) to act as a sort of external memory. That's very long term though.

> Doing the following gives me an Internal Server Error for some reason.

Should be fixed now

Make it available through a web page instead of a raw search dump?

Hide the internals of your search engine? In case you want to switch to meilisearch, algolia, ... (for cost reasons)

Preferably use your own search DSL to avoid users to learn about Elasticsearch queries (goes hand in hand with hiding internals of search engine)

Good luck! :)

If he changes the engine he might as well implement the Elasticsearch language then for the new engine.

$ rip() { curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode 'q=$@'}

$ rip heartbleed bug

  rip() { curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode "q=$*"; }

I still support the point that there is no reason for this to be a (grammar-defying) alias rather than a (tame) shell function or even a separate script.

Can you expose the filtering features of ES? I'd love to query e.g. "+python lists" and get results only related to Python, no e.g. Lisp results. For Stack Overflow you could use the question's tags as filter keys, and for other sites you'd add them manually (so e.g. the PHP docs get the PHP key).

If you're thinking of monetizing this, I'll tell you what I tell all the small, useful services that I'd like to pay for. There are too many small, useful services that I'd like to pay for. I'll gladly pay $1 for such a service, but you'll have a hard time convincing me to pay more.

It would be nice if the main page linked to your blog or anything really, because I would like to know where can I follow this project!

I'm giving this project a year to build up momentum. If it looks promising, I plan on having other STEM verticals. Maybe even fix recipe searches one day :)

A real homepage is coming. Feel free to subscribe to my blogs RSS feed for now: https://landshark.io/feed.xml

I'm guessing that's because it doesn't index docs.rs and the rust forum. Both incredibly important for Rust development.

So as long as this engine doesn't also index most programming related forums, I won't be able to use it effectively, even though I really would like to.

The concept of limiting the scope to just a few websites sounds really interesting, though. I think I will take this idea and build a little thing on top of google to implement that site filtering on my queries.

https://dontbeevil.rip/search?q=reddit

Not yet. I'm focused on explicitly developer-oriented resources right now. Those you mentioned are on the TODO list though.

apt install curl jq

pip3 install jtbl

curl -O https://raw.githubusercontent.com/alangibson/dontbeevil.rip/...

chmod u+x rip

./rip 'what is a monad'

  rip() {
      curl \
          --data-urlencode "q=${1}" \
          --get \
          --header 'Content-Type: application/json' --header 'Accept: application/json' \
          --silent \
          'https://dontbeevil.rip/search' \
          | jq '[ .hits.hits[] | { title: .fields.title[0], url: .fields.url[0], highlight: .highlight.text[0] } ]'
  }

$ rip() { curl -G -H "Accept: text/plain" --url https://dontbeevil.rip/search --data-urlencode "q=$*"; }

$ rip Heartbleed bug

edit: alangibson's solution in this thread is better :)

Also, am I correct in assuming it's not open source?

You'll be disappointed though as most of the important stuff only lives as BigQuery queries. I will be updating it in the near future though.

Edit: also some corporate technical documentation like Mozilla, Microsoft, IBM, etc have many such developer pages.

A next step is to index entire sites, not just individual pages, based on the positive votes their links get.

I'm considering opening up full ES query support for paying customers, but it's too dangerous to expose it to the Internet unrestricted.

If you do not already have a global rate limit, implement one ASAP. Better to have one and not need it, than to need it and not have it.

{"message":"Missing Authentication Token"}

tl;dr: https://dontbeevil.rip/search?q=monads&from=10

Stop attacking me!

In the mean time you can use application/json:

curl -G -H 'Accept: application/json' https://dontbeevil.rip/search?q=zelda

I will also be using the PageRank calculated by Common Crawl as soon as they release the next data set.

I'll put you down as a +1

> {"message": "Internal server error"}

There's many more web pages coming. They are much more difficult to get ahold of and rank though because I need to run my own crawler to fill in what Common Crawl doesn't have and then calculate my own site authority rankings.

  1. Stand up ElasticSearch instance
  2. Have it index SO and HN
  3. Charge $10 per month
  4. Profit!!

I'm spending over $200 per week just to stand up the service as it is. $10 for a full functional search engine will likely not be even close to PROFIT!!!!

What are using if you don't mind me asking? Not trying to criticize or anything. I have a Heztner box that gives me 1TB SSD in RAID 0 mode and 64 GB of RAM for about 80 CAD a month.

A big chunk actually goes to BigQuery. There are publicly available datasets for HN, Stackoverflow and a few others there. I've also loaded up the Common Crawl index. The query and storage fees really add up.

I'm hopefully done with huge BigQuery queries, so that $200 will probably drop for a while.

Maybe you're overkilling with the infra stack.

I would simplify until having a mature product, especially if I'm bootstrapping, which I think is your case.

Right now, you're still a bit far from MVP, from my point of view. Those $200 can probably be reduced by 50%-75% if you compromise on stuff only important to non-alpha services (i.e. 99.99% availability). A single EC2 box should be enough. Maybe look into Postgres or another FOSS instead of BigQuery.

These $100-$150 savings per week can go into promoting your service, getting as much attention as possible to maximize feedback.

Good luck!

Source: https://news.ycombinator.com/newsguidelines.html

So the difference is, it has the things folks on HN say they want: - search expressions - REST api - no tracking - users are buyers, not products

Great, now you just found a way for malicious actors to create reputation bombs, remove honest nodes from the pool and make it even easier for them to spam/poison the results.