It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping.
This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all companies. If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service.
There are many companies that sell data scraped from LinkedIn as a product. LinkedIn won a court case against hiQ Labs for scraping member data and other things[1]. I am not trying to compare this court case to the OP's website, just something worth mentioning.
In any case, web scraping is a sort of gray area of the law. In my opinion, this data set does not contain member data and is not being monetized so it feels kosher to me.
(Opinions expressed are solely my own and do not express the views or opinions of my employer.)
> In any case, web scraping is a sort of gray area of the law.
I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.
It reminds me of a similar conflict in public records.
Many municipalities charge for copies of public records. One can go to viewing rooms and examine the records at no charge. We all own those records as members of the public that funded them, and the place where they are kept.
Many municipalities want to prohibit photography because people taking their own picture of a public record does not involve the copy fee.
The municipality confuses access to records as a part of their fee, and that is where attempts to limit photography come from.
The fee actually funds the work necessary for an official copy to be made, optionally stamped to be admissible in court or accepted as an "original copy."
A photocopy of a death certificate is no different than the one the city clerk made, except for the stamp and clerk being able to testify about making the record copy.
Some companies and government agencies will accept a death cert no matter what. Others want an official copy.
Getting back to scraping:
Clearly people have access and can make their own data copies.
Maybe the answer is for companies to make official data products available, or something along those lines.
I believe that decision was that web scraping doesn't violate the CFAA unless there are access controls (such as the need to log in) to get to it, but they specifically said that the ruling doesn't comment about other possible claims against it.
That's why I think it's still a gray area. But I could be wrong -- I haven't been following this stuff all that closely.
You don’t need consensus to allow something, you need consensus to be against something, otherwise it’s allowed.
This “scraping is Grey” sounds like FUD to me. Legally, you can scrape anything publicly available as long as you damage the server. You understand that google is just a giant web scraper right?
And all the other search engines spidering the web. Do you think they are risking their business being in the “grey?”
There does seem to be some distinction if you log into and scrape:
"Only three claims remained for the final order - the violation of the CAN-SPAM Act, violation of the CFAA and California Penal Code.
The district court then granted summary judgment to Facebook on all three of the remaining Facebook claims. The district court awarded statutory damages of $3,031,350"
> In any case, web scraping is a sort of gray area of the law.
I feel like it's also only a matter of time before we have decentralized web scrapers that can consume data and assemble datasets and store the results in a decentralized fashion in a way that is completely non-enforceable. LinkedIn data would be aggregated in all sorts of interesting ways, and there would be literally no name you could sue, no address you could serve papers to, it would be a hundred thousand bots in 100 countries that collectively assembled the dataset and put it on some IPFS or torrent.
If data is visible with human eyes, it's visible by decentralized bots, the only missing piece is the technical and financial complexity of the current state of decentralized compute.
> If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service.
Well. This is certainly not true.
Even with Premium subscription you can hit a hard limit of personal profile views while just actively going through profiles without any automation. And Linkedin will temporary restrict and threaten to ban your account.
I have lots of LinkedIn pages for various companies (alive and dead). I’ve never had to pay: while there’s limits on page creation for free accounts (which appear to be prevent abuse) there’s no direct payment required to create a company page.
I’m not sure if you’re accidentally leaking a new strategy that LinkedIn are about to launch or if you’re mistaken but given you’re claiming to work for LinkedIn, probably worth correcting your comment either way.
The limit only applies if you’ve created… I can’t remember the number exactly and it’s not listed anywhere but I recall hitting a limit around 5 pages.
I’m on the same page. It also requires users to create an account in order to get the free version, and I would argue that in most cases that’s part of a monetization funnel.
The submitted title was "World's largest open source company dataset", but (1) "world's largest" is linkbait and the article walks it back, (2) "open source" could be worded better per https://news.ycombinator.com/item?id=35979581, and (3) the only thing left in the title after taking those out would be "company dataset", which is too generic to be a good title.
To be frank, we went back and forth on this, but in the end, thought the original title was ok. The only other large, "open source" dataset we could find was 9M. So after researching, we came to the conclusion that it sounded clickbaity, but was likely accurate.
And yes on "open source". We fully intend for this to be "open source" in the full meaning of the word, but it seems we were moving too fast and missed adding the formal license.
Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the past few months, and we're excited to finally share it with you all.
We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key company fields like name, industry, size, location, LinkedIn handle, etc. We aim to update it quarterly to ensure that you always have the most up-to-date information.
Disclaimer: Okay, we have to admit, we didn't exactly comb through every dataset out there to verify that ours is the world's largest, but we did our research, and we're pretty sure it might be. Whether or not that's true, we believe this dataset is a robust and invaluable resource for anyone interested in company data.
Data is very likely to be from LinkedIn if you look at the field descriptions and stats. The only field that is 100% available is the one based on the LinkedIn URL. I would guess scraping unless LinkedIn provides an API for this data that I can't find.
Most likely from scraping (crunchbase, yahoo,etc), unless they bought it from somewhere. In most countries you can get it from the chamber of commerce. Dun and Bradstreet and other similar companies. Some of these data aggregators will have partnership with other companies, and you can also (illegally) scrape it from there.
So this blew up today. Reviewing all the comments now.
1. Yes, this is scraped from public sources.
2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here.
3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but the website is not.
The data is not licensed as opensource (only non-commercial use): "The Service and its entire contents, features and functionality (including but not limited to all information, software, text, displays, images, video and audio, and the design, selection and arrangement thereof), are owned by the Company, its licensors or other providers of such material and are protected by United States and international copyright, trademark, patent, trade secret and other intellectual property or proprietary rights laws. You are permitted to use the Service for Your personal, non-commercial use, or legitimate business purposes related to Your role as a customer of BigPicture.io." - https://bigpicture.io/terms
It's open source in the sense of OSINT [0]. Clearly confusing on a site like Hacker News, but this has been standard usage of the term for that community for a long time now.
Thank-you for the clarification. "Open-source" is definitely different from "Open Source."
Meaning Open-source (sourced from open sources) but claiming Open Source is disingenuous.
Wikipedia isn't helpful either, because it refers to OSS as Open-source Software [1].
Open Source meaning may be more useful in comparison to Free Software. Stallman refers to "Open-source" (hyphenated) only once in this article, but only to refer to it as confusing versus free software [2].
It's possible "OSINT as Open-source" has been in use for longer than Stallman's use of "open source," but definitely they are different.
It's strange a site would sell up a feature on HN as "Open-source content, in the meaning of OSINT" without being up-front about it. The default assumption would be "open source as code that is free to modify, etc."
The mental gymnastics would be
1. They claim it is "open source."
2. They are talking about _content._
3. It must be the OSINT kind of "open."
This could be a pattern, because they're always needing to add another comment, "Just kidding, we meant OSINT open; we're not sharing the code."
... documentation could be open source too, though--in the sense of "free to modify, etc" and not "sourced from freely available data."
Could it be both? Only if they accept contributions, I guess.
No, the source code is not available. This dataset is a subset of the raw data our system collects. Our final product made available via the API does a variety of processing steps on the raw data (dedupes, joins, ML predictions, etc). The final, processed data is the piece that is proprietary / subject to the terms.
We will update the site terms to reference this dataset, as we aim to continue releasing an updated version each quarter. I'll have to double check with the lawyers, but it will most likely be MIT licensed.
What exactly does "open-source" mean to you? Because it sounds like there's absolutely nothing open about this other than a small scraping of LinkedIn data (which you should probably ask your lawyers if you're even allowed to license out).
The wording isn't just misleading, it's a complete lie.
EDIT: Nevermind, the title has been updated to accurately reflect this being a small datadump.
Ok. I'm not sure how this happened, but I think the dataset was somehow mislabeled. It appears that this dataset is the Q1 version, not the latest Q2. Can you please try re-downloading it?
We're probably going to have to make an public announcement about this...
269708: London
124059: Paris
113135: New York
99314: São Paulo
75428: Los Angeles
69691: Madrid
67328: Toronto
63738: Dubai
63456: New Delhi
I would not have expected São Paulo to come fourth in this list, after New York but in front of Los Angeles. I just learned it's the 4th largest city in the world https://en.wikipedia.org/wiki/List_of_largest_cities - after Tokyo, Delhi, Shanghai - but I guess it has much more of a representation on LinkedIn than those other cities.
I've been working through this issue a lot lately. There tend to be two camps, the "Make the experience good for the person trying to access the data, because good relationships are more important than the ability to contact someone" versus "Why should I give someone something for free when there is zero chance I can ever make a sale to that person?"
This whole thing is amateur to the core, it’s an improperly licensed (are you open source or not? What license? You don’t get to say “probably eventually MIT” lol) freemium scrape of linkedin masquerading as a product
That is literally their own site. I don't think they would DMCA themselves
Thanks for the link. The work email requirement for just the download is just bullshit (yes I did read their piece, with that kind of logic they could also require credit card info etc while still calling it free like many scammy sites do)
Good luck with your launch! This reminded me of a similar project, the opencorporates database (https://opencorporates.com/), though the target use cases seem different.
"With over 15 million global companies included..."
What distinguishes a global from a non-global company? Also, how many of these are anonymous Delaware/Nevada/South Dakota/etc-based shell companies, or are those excluded from the dataset somehow?
On your home page you list Microsoft as being one of your clients. I'm pretty impressed that you managed to sell them what appears to be (mostly) their own data.
We went through this in the ad world before GDPR existed. Companies were selling datasets for advertising and, I remember one of my clients (one of the largest ad agencies in the world) telling me they had over 100 sources of data they augmented their publisher data with but they tested and saw there was only about 20% uniqueness on average between sources.
They were buying the same data again and again, recycled and repackaged.
The moment those graph connections will show negative view of some of those CEOs they will start screaming about privacy issues and/or sue the graph creator.
With scraping LinkedIn alone it should already be possible to create such graphs. Not sure why I haven't seen any. Either scraping LinkedIn isn't as straightforward or such grap-creating attempts have been shut down due to potential for defaming someone.
I looked at the attributes they say the dataset has.
Not too many (e.g. number of people, location).
The really interesting ones, like who is doing business with whom, are missing.
How hard is it to scrape LinkedIn for all its public profile data? Do you need special developer access? Do you need to sign some contract with MSFT for anything nontrivial?
Apologies. We've had a huge problem with bots, so we have a number of security measures in place. The Google Captcha component is probably flagging you as a bot.
Try disabling your VPN if you're on one, or use a different IP.
This is awesome and in the ballpark of something I'm working on right now.
I currently have a list of developer handles, their associated aliases, and their associated email addresses - trying to map that set to employment history.
How do you plan to identify or differentiate between legal entities? Eg a big company like your Uber example will often numerous subsidiaries, in many jurisdictions.
Do you plan on including well-known identifiers like LEIs in your model?
I searched for a dataset like this for a long time, trying to use it to augment named entity recognition tasks for documents. But now that GPT is on the market, this works out of the box. It's still useful as reference for validation.
The blog picture looks to be generated with dalle2. The quality was mind blowing less than a year ago, while it is now a mess of artefacts compared to dalle2.5 (Bing), adobe firefly, stable diffusion, and of course MidJourney.
Individual countries know this, and even share rather successfully across unions.
EU exchanges VAT information which is exactly about who buys from whom... down to transactions of 120€ and more if I am not mistaken.
Commercial databases are more limited, but even those can trace ownership of companies to ultimate beneficial owners. Orbis from Bureau Van Dijk is probably the biggest of such databases.
This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all companies. If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service.
There are many companies that sell data scraped from LinkedIn as a product. LinkedIn won a court case against hiQ Labs for scraping member data and other things[1]. I am not trying to compare this court case to the OP's website, just something worth mentioning.
In any case, web scraping is a sort of gray area of the law. In my opinion, this data set does not contain member data and is not being monetized so it feels kosher to me.
(Opinions expressed are solely my own and do not express the views or opinions of my employer.)
[1] https://www2.staffingindustry.com/Editorial/IT-Staffing-Repo...