It sucks to say, but maybe the best solution is to pay Cloudflare or another fancy web security/CDN company to get rid of this problem for you?
Which is extremely dumb, but when the alternatives are “10x capacity just for stupid bots” or “hire a guy whose job it is to block LLMs”… maybe that’s cheapest? Yes, it sucks for the open web, but if it’s your livelihood I probably would consider it.
(Either that or make following robots.txt a legal requirement, but that feels also like stifling hobbyists that just want to scrape a page)
> Either that or make following robots.txt a legal requirement [...]
A legal requirement in what jurisdiction, and to be enforced how and by whom?
I guess the only feasible legislation here is something where the victim pursues a case with a regulating agency or just through the courts directly. But how does the victim even find the culprit when the origin of the crawling is being deliberately obscured, with traffic coming from a botnet running on exploited consumer devices?
It wouldn't have to go that deep. If we made not following robots.txt illegal in certain jurisdictions, and blocked all IP addresses not from those jurisdictions, then there would presumably have to be an entity in those jurisdictions, such as a VPN provider, an illegal botnet, or a legal botnet, and you pursue legal action with those.
The VPNs and legal botnets would be heavily incentivized to not allow this to happen (and presumably already are doing traffic analysis), and illegal botnets should be shutdown anyway (some grace in the law about being unaware of it happening should of course be afforded, but once you are aware it is your responsibility to prevent your machine from committing crimes).
Illegal botnets aren't new. Are they currently shutdown regularly? (I'm actually asking, I don't know)
> If we made not following robots.txt illegal in certain jurisdictions, and blocked all IP addresses not from those jurisdictions
That sounds kinda like the balkanization of the internet. It's not without some cost. I don't mean financially, but in terms of eroding the connectedness that is supposed to be one of the internet's great benefits.
Maybe people need to add deliberate traps on their websites. You could imagine a provider like Cloudflare injecting a randomly generated code phrase into thousands of sites and making sure to attribute it under a strict license, that is invisible so that no human sees it, and changes every few days. Presumably LLMs would learn this phrase and later be able to repeat it - getting a sufficiently high hit rate should be proof that they used illegitimately obtained data. Kinda like back in the old days when map makers included fake towns, rivers and so on in their maps so that if others copied it they could tell
But you needed a private entitlement for it to work that no app (save UTM after a jailbreak) had, so the pet theory at the time was internal Apple engineers were running around with Linux on their iPads.
I think like a lot of things, once you’re used to having the knobs of k8s and its DX, you’ll want them always. But a lot of teams adopt k8s because they need a containerized service in AWS, and have no real opinions about how, and in those cases ECS is almost always easier (even with all its quirks).
(And it’s free, if you don’t mind the mild lock-in).
While you can do that, you lose the flexibility of IIS without a lot of infrastructure development should you not just want a single website per server running .NET only.
Yes, that is obvious, but the problem is that this requires some account to have permissions to start and stop services and to execute commands on the target host. Corporate IT departments are not too happy with that kind of approach nowadays.
For the first example, Microsoft might point a finger during a audit[1] but if the military had a license and just used a cracked version instead to bypass the activation prompt, I don't think they'd really make a fuss? If it turned out that they were only buying 50% of the licenses they should've been then sure, but if they were buying them and just not typing the license keys in, meh.
I seem to recall the network activation server for Windows let you activate unlimited clients on a key, although maybe I'm misremembering, or the person who configured it did something funny.
---
[1]: Does Microsoft even do first party license audits anymore? I'd imagine with so much of the licensing being a part of cloud spend that they don't care since you're already paying $20+ per month per head anyway for windows and office.
For the US Military pirating software while on deployment in Iraq, Microsoft would need to seek redress in Iraq's court system.
28 U.S. Code § 1498 [1] holds the federal government liable for copyright infringement (section b), but only in the US (c).
I don't know how much copyright enforcement Iraqi courts were doing during the occupation, and the US Military was operating with broad immunity from Iraqi law as well. There's no reason for Microsoft to make a fuss over this infringement, because they're not going to get compensated and the PR will be negative.
When the company I worked for for 20 years was sold to a muuuch larger company, people from the new company were telling us how they just finished a big audit from MS and they did well and they were so happy they came out more or less clean. Not just relieved they made it through a trial that some douchebags imposed on them, actually pleased with themselves for pleaseing MS is what it almost felt like.
I was boggled that anyone would be happy about any part of that.
I had successfully avoided letting us ever rely on any MS services other than most people's desktops were Windows.
I can't even imagine so much as letting MS or anyone else even in the building to go through our shit to satisfy their desire to know if any of our machines were running any of their software. I'm sure it's part of some license agreement that you agree to be subject to audits or else you don't get to buy the stuff at all but still.
I can't imagine being pleased that some douchebags have audited me and said I was a good boy this year.
As a linux user it boggles me what people are willing to put up with and consider it not only normal but even valid.
I'm not full Stallman and I will use proprietary stuff when I have to, and deal with it's terms, but I never even slightly actually internalize the vendors ideas about what I should do.
What does value in the product have to do with audits?
It makes no sense to be happy about audits, even successfully passing them, regardless how much value you get out of whatever product they are attached to. The two things have no bearing on each other. By all means, value the product greatly.
This excusing is exactly what I was talking about. It's kind of disturbing and sick even to be so cooperative with a bully.
It's one thing to do what you gotta do and put up with what you gotta put up with, but to actually not see a problem and derive personal sense of worth from some shop owner going through your pockets to satisfy themselves that you didn't steal anything is just bizarre. You should not be happy that the shop owner said "looks like you didn't steal from me today".
Not everyone is a Linux user, or willing to become one.
You can install Ubuntu on 100 Thinkpads and have that be the official OS of your company.
The tech folks will love it, but Mr Smith in accounting will be angry and confused. QuickBooks doesn't work, how will he get anything done ?
He doesn't want an open source alternative. He wants QuickBooks.
This is absolutely on point. Every time Microsoft comes up, HN thinks you can just replace an orgs 30 year old stack with OSS. It's ridiculous and shows the narrow experience of this crowd with their small teams.
Linux still doesn't have a lot of one to one equivalents for Microsoft software. If Mr Smith in accounting likes using QuickBooks and Excel, you can't just tell him it's time to switch to libre office.
I definitely think Linux is better for personal usage, but I don't expect most companies to ever realistically roll it out to every desktop.
> I was boggled that anyone would be happy about any part of that.
Would you be relieved to have passed an IRS tax audit? I know I would.
Yes, it is part of the license agreement to be audited, but there are two kinds of audits -- the one you can tell them to go pound sand, these are usually the ones from 3rd parties, and the ones that have the license agreement enforcement behind them.
> As a linux user it boggles me what people are willing to put up with and consider it not only normal but even valid.
Corporations are because of the the MS stack that still has no equal in AD DS, client management, etc. It has gotten easier with cloud licensing since you can't usually go above the license count you ordered.
which is equally funny I think