Hah, I literally just fought this for the past month. We run a large esports league that relies on player ranked data. They have the data, and as mentioned above, they send it down to the browser in beautiful JSON objects.
But they're sitting behind Cloudflare and aggressively blocking attempts to fetch data programmatically, which is a huge problem for us with 6000+ players worth of data to fetch multiple times every 3 months.
So... I built a Chrome Extension to grab the data at a speed that is usually under their detection rate. Basically created a distributed scraper and passed it out to as many people in the league as I could.
For big jobs when we want to do giant batches, it was a simple matter of doing the pulls and when we start getting 429 errors (rate limit blocking code they use), switch to a new IP on the VPN.
The only way they can block us now is if they stop having a website.
Give one of the commercial VPN providers a try. They're usually pretty cheap and have tons of IPs all over the place. Adding a "VPN Disconnect / Reconnect" step to the process only added about 10 seconds per request every so often.
It probably doesn't save you much, since you already built the chrome extension, but having done both I found that tampermonkey is often much easier to deal with in most cases and also much quicker to develop for (you can literally edit the script in the tampermonkey extension settings page and reload the page you want it to apply to for immediate testing).
I might be wrong, but some sites can block 'self' origin scripts by leaving it out of the Content Security Policy and only allowing scripts they control served by a CDN or specified subdomain to run on their page. Not sure when I last tried this and on what browser(s).
You'd have to disable CSP manually in your browser config to make it work, but that leaves you with an insecure browser and a lot of friction for casual users. Not sure if you can tie about:config options to a user profile for this use case. Distributing a working extension/script is getting harder all the time.
I don't recall if I've encountered that specific problem in tampermonkey (or if I did and it didn't cause a problem worth remembering), but you can run things in the extension's context as well to bypass certain restrictions, as well as use special extension provided functions (GM_* from the greasemoney standard) that allow for additional actions.
I do recall intercepting requests when I used a chrome extension to change CSP values though and not needing to when doing something similar later in tampermonkey, but it may not have been quite the same issue as you're describing, so I can't definitively say whether I had a problem with it or not.
Holy Moly! The author of this paper makes a very scary argument further shifting the burden of proof onto the employers.
The problem with hiring is that there is a hugely disproportionate supply/demand ratio. We might have 1 position to fill and 100+ applicants. How can any company correctly protect themselves against allegations of illegal hiring criteria in this type of environment? Right now the best protection is to say "we hired the most qualified candidate", but even that argument is shaky. What if the most "technically qualified" candidate performed the worst in the interview process?
Hiring is scary enough already without bringing the law into the picture. :/
Well, the law is already in the picture; discrimination lawsuits aren't a new thing. And it's also true that, just because you encode certain things in an algorithm (or just let an algorithm learn from existing discriminatory practices), you can't absolve your own responsibility.
There are practices for employment advertising/recruiting that certainly highlight issues in a new way. Is doing on-campus recruiting or advertising in media that targets specific young demographics illegally discriminatory? Probably not. (IANAL) Is targeting young people using Facebook algorithms OK? Dunno.
> Hiring is scary enough already without bringing the law into the picture.
This is at the root of a lot of social issues lately.
One person, let's say it's a person of colour, is afraid of not being able to get a job at all.
Another person, who is doing the hiring, is afraid of someone holding them accountable for actually hiring qualified applicants.
This is a very asymmetrical pair of concerns. But if you're doing the hiring, it feels like the worst thing in the world to have someone second-guess your decisions.
You are so busy worrying about this that you don't even consider how scary it is to look for a job, and have robots tell you that you aren't getting an offer because your diction and facial expressions don't match those of people who the company has already hired.
If the hiring process is transparent, wouldn't job seekers just figure out how to game it? Unless the process is confidential, and only comes out in lawsuits, and isn't disclosed to the public, this would only serve to further obscure the candidates who can do the job as opposed to just get hired.
They aren't really trying to hide it, they send you an email with a page or two full of topics and links to help you prepare for the interview. The current state where people practice for their interview is exactly what they want.
Yes. But that's already the case and is usually a useful screening criteria anyway. i.e. I don't actually care if you smoke pot or not. But I care whether or not you are smart enough to lie about it during an interview or job screening process, and if you can quit long enough to pass a drug test. Because that's relevant.
If you are intelligent enough to game the system into looking like you meet all of my criteria, you can probably also "game the system" into actually meeting my criteria on the job too.
That would seem to have a major chilling effect on hiring practices across the board. For what it's worth, the reduced friction for companies in the hiring process, i.e., not having to explain every decision, moves the process along fast enough to keep the economy churning. I'm not sure that would be the case if all that bureaucracy were installed into the process.
The point of a transparent process would be to make the rubric as non-subjective as possible. A "how much they like me" criterion, if it were still part of the decision, would become very obvious if/when a company selected some candidate that was not in fact "the best" according to their transparent rubric.
But how do you put team fit into a rubric. It's absolutely an element - a team of 1Xers will be more productive than a team or 10Xers who all can't stand each other. I wouldn't want to be part of a team where everyone has an excellent objective mark, but is just a bunch of jerks. What part of the objective rubric accounts for "can't play well with others"?
One way is to define a set of bullet points that are representative of the team’s culture. This isn’t always easy, but is a worthwhile exercise even outside the context of hiring. Think: a localized version of the Joel Test. These values can be objective (“we strive for 100% test coverage”) and subjective (“we value communication over process”). Every candidate can be ranked relative to these values. The most important factor here is to rank every single candidate against every single value. This will most likely lead to additional questions for some candidates, such as, “tell us about a time when you found communication or process more helpful than the other, and how it made you feel”.
It's easy to frame "team fit" as "a team of 1Xers will be more productive than a team or 10Xers who all can't stand each other," but I am going to put Fleetwood Mac's "Rumours" album on and say "citation needed."
Also, "team fit" is very nice when used in good faith, but all-too-often, hiring for "culture fit" or "team fit" ends up being a euphemism for "valuing a monoculture over valuing competence."
I may have exaggerated with the values, but if we can't both agree that people who work well with each other are more productive than those who don't, I don't know what to tell you.
While it may be used poorly, it does still have value, and that's why it exists at all. If judging someone's personality and attitude wasn't important, there'd be no need for in-person interviews at all.
Well, you've already reframed the point you're trying to make. You just said "people who work well with each other are more productive than those who don't," but that's tautological, the very definition of "people who work well together" is that they are productive, and the definition of "don;t work well together" is that they aren't.
Whereas, in the comment I was responding to, you talked about "people who can't stand each other." You haven't made the case that if people can't stand each other, then they necessarily can't work productively together.
Furthermore, you seem to imply that if you can learn something about someone's personality and attitude, then you can judge whether they will be productive working with other people.
That's a perilous endeavour. It may seem intuitively obvious, but it isn't by a long shot. There are lots and lots and lots of examples of people who don't like each other or enjoy socializing or joking with each other who nevertheless get things done working together.
Furthermore, when people set out to build a framework around interviewing someone, or asking them a bunch of questions and from there working out their "compatibility" with other people, we always end up with something like Myers-Briggs.
Thaat kind of thing has been conclusively debunked as a signal for hiring or not hiring people. But most people aren't even as quasi-objective as Myers-Briggs. Most people just go with their intuition, which they couch as "experience."
When you ask such people, they tell you they are very good at hiring people. But of course, they don't really know that in an empirical sense. They don't measure their false negatives, they have no idea how many people they disliked turned out to be productive workers.
They don't work from a reproducible framework. In the end, they're just working their own biases and crediting their expertise for all the successful hires while downplaying unsuccessful hires, and ignoring outright no-hires who turned out to be successful elsewhere.
At the end of the day, you're running a business or a movement or whatever, and people have to put aside their likes and dislikes and execute on the mission. Most people can do that just fine. There are some exceptions, people who are toxic, but honestly those are the exceptions.
I find, with my n=1 sample set, that when you hire for competence, and hire people who have demonstrated competence in the past, they almost always get along well enough to do their jobs.
And when they don't, there's a little thing called management that you use to get people back on track. In my n=1 experience, people who have competence out the yin-yang but are so toxic that they cannot be managed and must be fired are rare, and easy enough to detect in the normal course of interviewing that there ois no need to constantly ask myself if the person I'm interviewing can get along with the team.
If they can make it through the interview process without insulting everyone or having a temper tantrum, and if they demonstrate competency programming, designing, and communicating their technical rationale, my experience is that they're nearly always going to get along well enough to G$D.
> the very definition of "people who work well together" is that they are productive, and the definition of "don;t work well together" is that they aren't.
I don't agree.
Working well together suggests they cooperate to reach a common goal, regardless of their individual productivity.
Not working well together suggests that they might even waste their time attacking each other instead of working towards a common goal, which has a negative impact on a project regardless of how productive each individual might be.
The Disparate Impact Doctrine doesn't allow adverse effects on that could cause unintentional discriminatory impact for protected class minority groups. This law exists for decades before algorithmic decision making become a fad. In case of supply/demand ratio arguments, this doctrine takes into consideration the "outcomes" relative to the input. That is, if you are able to give a reasonable explanation for your outcomes - you wouldn't be labeled illegal.
If the most qualified candidate bombed the interview, they wouldn’t get an offer this time, but probably would in the next interview when their performance reverts to their average level of competence.
Some companies place rejected candidates on de facto blacklists to avoid wasting time with a recruitment process that's likely to be rejected.
This is particularly common when the hiring process is delegated to third-party HR/recruitment services, where recruiters are averse to the possibility of appearing ineffective in the eyes of their client.
That’s that company’s choice of how to conduct recruiting. Other (perhaps more competitive) companies say “you can re-apply after N months.”
I have a hard time seeing that as improperly discriminatory any more than departed employees having an “eligible for re-hire” bit set to false.
“Plenty of other fish in the sea” works both ways.
Maybe turn the question around. If someone bombed the interview, is there some additional process a company should go through to ensure they aren’t in fact the best candidate? What form might that type of inquiry take and why wouldn’t we just use that instead of the interview process for the company’s purpose in selection?
I've noticed the same thing and have been actively working over the past 5 years trying to make introductory robotics approachable.
I've been a software developer for a long time, but robotics always felt like a mystery to me until I had someone show me the basics. I recently started a small company trying to teach the basics of robotics. The biggest challenge is that robotics have a real price attached that normal software development doesn't have associated.
With Javascript, I just need a text editor and a browser, but with Robotics I have to make the leap and purchase a kit and trust that the kit will do enough to teach me what I need to know to get started.
We designed and built a low-cost kit and have been using it in middle-schools throughout the southeast teaching kids the introduction, and so far that's been working really well.
Unfortunately it seems like it's really hit or miss depending on which worker handles the order.
Some days I get exactly what I ask for really quickly, when other days I'll have my shopper switching 25% of my requested items with alternates (some of which are wildly off base).
My biggest frustration is that it seems like the shoppers aren't intimately familiar with my local grocery stores. I'll request an item that I know exists, they'll swap it out for something completely different and not what I want, yet if I personally go to the store, I'll easily find the item I requested.
One cool approach to a solution would be to use data to save the day. Millions of people fly through the carriers each year. The airlines could analyze the statistical heights of their average airline passenger and design the seats accordingly. Then assign seats on the flight based on passenger height. You could have a percentage of your seats configured for short people, a bigger percentage configured for average height, and several set up for tall outliers. Then everyone will be statistically more likely to get a comfortable experience.
Twitter has caused serious harm to myself and my family. I'm a big fan of the platform, but I've been "doxxed", harassed, and was even SWATed once by different people trying to get me to give them my username.
I'd release it, except that'd be almost as big of a pain. I'd have to release a new edition of my book, change all my websites. Ugh. People are dicks. I wish I'd picked a less desirable handle.
We used them to design and build a custom board for our Sumo robots that had our V1 built from an Arduino board and a custom board that plugged into the Arduino. We couldn't really afford to pay them a big chunk up front, but we negotiated a pretty pleasant royalty deal that comes out to a few dollars per board (which we get manufactured for less than $10/board).
They did excellent work and I highly recommend them for that type of "prototype -> production" path.
Doesn't this use case eventually screw over any future Twilio customer that selects that number? I see several inbound calls to my Twilio number from random numbers that I have to pay for, even though I'm only using Twilio in a development capacity right now. Do you guys "expire" numbers from your pool that receive aggressive amounts of Inbound traffic (voice, fax, or SMS)?
At some point a few years back, my VW dealership sold my phone number to people for vehicle service contracts. Even though I sold that car 4 years ago, I STILL get at least 2-3 calls per week from different companies offering to sell me an extended warranty. How does Twilio mitigate that for us?
(Off topic, but seriously, thanks for Twilio. Ping me if you want to know how we're using it in our new startup [email in profile]. It's pretty rad.)
Just wrote a lengthy reply to this concern up above, but the tl;dr is that we sit on all numbers for at least 2 months and wait until there's an acceptably low amount of inbound traffic before releasing them back into the available number pool.
Would love to hear how you're using us. Will drop you an email. (And thank you!)
Do the numbers being held register as disconnected to anyone calling them? It seems like that might cut down on some of the junk calls by the time they're released.
It absolutely does. Twilio should provide usage history on the phone #. The best predictor of future undesired/spam calls is prior call volume. The use case described sets up terrible downstream effects.
A number having high usage history does not necessarily mean it still is being used. Providing these numbers to the customer is a neat idea, but since they only provide an indicator for how it _was_ used they won't have much value. It might also be a bad PR move since users will draw conclusions from that history that might not be true.
But they're sitting behind Cloudflare and aggressively blocking attempts to fetch data programmatically, which is a huge problem for us with 6000+ players worth of data to fetch multiple times every 3 months.
So... I built a Chrome Extension to grab the data at a speed that is usually under their detection rate. Basically created a distributed scraper and passed it out to as many people in the league as I could.
For big jobs when we want to do giant batches, it was a simple matter of doing the pulls and when we start getting 429 errors (rate limit blocking code they use), switch to a new IP on the VPN.
The only way they can block us now is if they stop having a website.
Give one of the commercial VPN providers a try. They're usually pretty cheap and have tons of IPs all over the place. Adding a "VPN Disconnect / Reconnect" step to the process only added about 10 seconds per request every so often.