Slightly broader question: Do you feel like there is any ethical considerations one should think about before using something like this?

Absolutely agree there are ethical considerations with web browsing AI in general. (And the whole general ongoing shift from using websites to using chatgpt/perplexity)

People are already deploying tools like Anubis[1] or go-away[2] to cope with the insane load that bots put on their server infrastructure. This is an arms race. In the end, the users lose.

[1]: https://anubis.techaro.lol

[2]: https://git.gammaspectra.live/git/go-away

E.g. if I'm the developer of a workforce management app (e.g. https://WhenIWork.com) I could deploy BLAST to quickly provide automation for users of my app.

I think it would take more substantiation to claim this. Maybe 10 out of 1000 websites will get closed, but the users will be able to use AI tools to use the remaining 990. Not sure about you, but sounds like a win for users to me.

this may be true for the time being (or not), but will sure change if/when more [websites] become aware of what is going on. The result will be 10 out of 1000 websites will remain open, and not the ones you actually want. The more pressure there is on the sites/servers, the more these will have to act to stay online to begin with.

A few vocal people on HN barely scratch the surface.

I'm personally not sure there are, but I'm curious to hear what those are for you :)

Unfortunately, it seems like browser-use tries to hide that it's controlled by an AI, and uses a typical browser user-agent: https://github.com/browser-use/browser-use/blob/d8c4d03d9ea9...

I'm guessing because of the amount of flags, you could probably come up with a unique fingerprint for browser-use, based on available features, screen/canvas size and so on, that could be reused for blocking everyone using Blast/browser-use.

If calebhwin wanted to make Blast easier to identify, they could set a custom user-agent for browser-use that makes it clear it's Blast doing the browsing for the user.

But the thing is, the more they push for this, the less accessible websites will get, so there has to be another solution. Not everyone uses a mouse.

Source: I work in the industry.

I don't have API for _that_.

anti-bot fingerprinting exists to prevent you from using the human-facing site like an API

BLAST can also be used to add automation to your own site/app FWIW.

I think it's cool that you're experimenting in this area, but I'm not a huge fan of this as answer to a question about responsible/respectful web crawling. This stuff seems like it should be table stakes (even if you wanted to make it optional for the end user), but "yeah probably; learn the codebase, fork it, make changes, then we'll review it" really puts the onus onto the original poster.

I'm really not excited at all about the "scrape other people's data" use case for BLAST and if we can prevent it then awesome. I'm excited about BLAST automating science, legacy web apps, internal tools, adding AI automation to your own app, etc.

Ad blocker is the least user can do.

One use-case for this is conversations: So for example if I invoke /chat/completions with [{"role": "user", "content": "Go to google.com"}] and later with [{"role": "user", "content": "Go to google.com"}, {"role": "user", "content": "Search for gorilla vs 100 human"}] then we cache the browser state from the first invocation so it can be quickly restored (or reuse the browser if not evicted).

Caching will get much more sophisticated in a future version, it's the piece we're most actively working on.

Though ultimately I think the web needs something better than MCP and we're actively working on that as well.

the next step is something like worldcoin/ID or biometric-gated access, or an extreme increase in the number of paywalls (not that I like either of those options). I don't see any other way forward on the decade+ timescale.

(I make AI agents as my day job, among many other things.)

Now the API is what may be throwing folks off. Right now it's an OpenAI-compatible API. We will implement MCP. But really the core thing is abstracting away optimizations required to efficiently run browser+LLM.

while LLM("is <task_description> not done?"): Browser.run(LLM("what should the browser do next given <Browser.get_state()>"))

This simple loop turns out to be very powerful, achieving the highest performance on some of OpenAI's latest benchmarks. But it's also heavily unoptimized compared to a system that is just LLM("<task_description>") for which we already have things like vllm. BLAST is a first step towards optimizing this while loop.

A queue? What else can you really do. Your server is at the mercy of OpenAI, so all you can do is queue up everyone's requests. I don't know how many parallel requests you can send out to OpenAI (infinite?), so that bottleneck is probably just dependent on your server stack (how many threads).

There's a lot of language being thrown out here, and I'm trying to see if we're using too much language to discuss basic concepts.

Now you are right that at some point you'll get throttled either by LLM rate limits or a set budget for browser memory usage or LLM cost. BLAST's scheduler is aware of these constraints and uses them to effectively map tasks to resources (resource=browser+LLM).

You wanna load test the local DOM rendering or what? Otherwise, whatever endpoint is serving the HTML, you configure your load tests to hit that, if anything. Although you'd just be doing the same testing your HTTP server probably already doing before doing releases, usually you wanna load test your underlying APIs or similar instead.

https://blast.ncbi.nlm.nih.gov/Blast.cgi