Great product. It should have happened last week, then we truly could have said that all Enterprise Apache Open Software products died of the AWS plague in 2018: Hadoop is gone with Cloudera/Hortonworks merging, Kafka with Confluent's new license covering their ecosystem of tooling. Databricks now have their own Spark runtime, which will get more TLC than Apache Spark. On top of Apache, Mongo, Neo4J, Redis Labs abandoning open-source. Wow, just wow.
We clearly say in all announcements that this is about doubling down on open source. In contrast to all current the efforts everywhere that restrict open use, we plan to do even more work openly.
This is best thing that could have happened to any open source project. And to any founder whose heart beats for technology and open source...
Aha, i see why you downvoted me. You were actually serious. I literally thought this was a generic tongue-in-cheek ironic comment made for the purpose of comedy. That's why it made me smile. That it's not ironic is even funnier.
You can’t downvote people who respond to you on HN, at least at his karma level. In fact, he couldn’t have downvoted you because he has too little karma.
Curious where you get "Hadoop is gone" by two companies (who employ hundreds of open source contributors and committers) merging. I think it's in fact the complete opposite. I get the whole buy-in into the FUD that Cloudera ships proprietary software so they're evil, but the fact of the matter is their core system (CDH) is 100% open source. Impala, open source. Kudu, open source. Anything that is in CDH that isn't in Apache Hadoop is in a pull request. You can literally download every last source file that you trust your data with.
Hadoop is not gone yet. But the intention is there. AWS EMR is killing them. Users/Companies just pick EMR by default, leaving few crumbs left for the now one company that develops the platform. The EMR team is what, a two pizza box team? Like you said, Cloudera pay for hundreds of top developers to develop a product that is repackaged by a two-pizza team whose company earns more than Cloudera on Hadoop. That situation will not continue, that is my prediction. And Hadoop will be the crossfire. My prediction is that Cloudera will release a new data platform in time that is not "hadoop", but they will leave Apache Hadoop in maintenance mode. It's the only logical way to address the AWS "problem".
That's where we disagree. You're taking some pretty radical jumps to get to the conclusion you want. Let's assume for a second that you're right, and EMR somehow kills an entire company (who sells much more than just a Cloud offering). This has absolutely no bearing on the survival of an open source project. Yes, Cloudera employs developers who's sole job is to work on Hadoop, but there are a ton of people who work on and contribute to Hadoop outside of both Cloudera & Hortonworks (e.g. Yahoo, Facebook, Amazon, Alibaba, ..., ..., ...)
It's equally outrageous to assume that the developers at these companies would somehow just completely lose all passion and willingness to carry the project forward if they lost their job. The reason a lot of people at Cloudera & Hortonworks are there is because they were contributing to the project long before they were employees.
Hadoop isn't going anywhere. It may become less relevant or more relevant based on how things play out and/or new projects coming to fruition, but the project will be around for a long time. Lastly, the project "Hadoop" is nothing like it was back 8 years ago. The project continues to evolve and projects born out of it are essentially tacked onto it (e.g. Flink, Spark, Kudu, HBase, Presto/Drill/Hive/Impala).
What is it about European Enterprise Data software companies that they never make it past this stage without either being bought up or moving to the States?
-Europe is fragmented market. It is not given that German software will be gladly accepted and used in France, Italy or UK and vice versa. There are barriers there that does not exist in the US.
-The capital market and the stock market are not as developed as in USA in terms of market participants and available funds. Going public is unlikely event. If it happens, it would not get a lot of publicity and it would not raise as much money. Some European companies prefer to go public in the US.
-There is public mistrust about big software companies. Many people are afraid that big software companies will steal and abuse their personal data.
-There is also mistrust about venture capital and generally big finance institutions that they somehow "control" the economy, exploit small family businesses, and just make money on mistreating normal people.
I know that some of this sounds ridiculous, but it is true for astonishing number of people in Germany, France, Austria, and Italy. UK is a little better, but they mistrust everything about the rest of Europe.
> Many people are afraid that big software companies will steal and abuse their personal data
Cant say they are wrong. But its also a business concern for enterprise companies everywhere - US or EU so you cant really say this is an issue here. More of a requirement.
>exploit small family businesses, and just make money on mistreating normal people.
Most of the people who think so are being very stupid.
> Most of the people who think so are being very stupid.
VCs make a profit by selling startups to big corporations or big investors via IPO. Most startups have negative cash flow and make negative profits by the time they are acquired by a corporation or investors.
This means that VCs make a profit by creating negative value. Most VC-funded startups with a successful exit never become profitable.
Those few startups that do manage to IPO and eventually become profitable are usually those who happen to have a monopoly over some industry; in other words, they are only profitable because consumers have no other alternatives to choose from; so they get to set the price.
Believing that VCs create negative value is not stupid. Not seeing that it might be the case is what's stupid.
I see what you mean. But the actual point was about exploiting family businesses and mistreating people. I am not sure what it has to do with your negative value argument.
One reason is because I think there isn't such an obsession with becoming a billionaire here. If you're a founder, and someone offers you a deal which will personally see you net €10-20m after tax, why wouldn't you take it?
I agree. However, I know the Data Artisans founders personally. There like like 6 or 7 of them, and Prof Volker Merkl (main person behind the original project, Stratosphere). Then, they did two funding rounds. Nobody is coming out with near 10m euro. Half that, before tax, probably after a few years of golden handcuffs. So, my guess is 3m euro after 10+ years. Not bad, but you couldn't buy a good house in the valley or SF for that.
That seems pretty low to me. Can you buy a house in Berlin with that income? It's almost what a dev makes in Romania and there you can buy a decent house with that income.
It is normal in Germany & even more so in the big cities like Berlin for people to rent their whole lives - there are a lot of tenants protections & rent control here that make that less risky.
So many people renting their whole lives sounds to me like in Germany most properties are owned by a privileged few and the remaining being forced to be their tennants. Does nobody see anything wrong with that? Rent control laws don't make this OK in my book.
If you're asking for my opinion residential real estate should be publicly owned, like hospitals, transport infrastructure, (public) schools/universities, etc.
Or at the very least the land under it should never be sold, only leased from the state.
Yes, consolidations on the market are a thing with an additional side effect - as owning a property is professonalized it’s even more difficult and expensive for an individual to own.
Having talked with some Romanians, software professionals’ salaries in Romania are not far behind average German ones. They wouldn't be the first I would accuse of wage dumping (that would be Ukrainians, Russians, Bangladeshis, and Indians). It’s that everything else in Romania is shit.
Except the nationalities you listed are not EU while Romanians are.
Thre's plenty of things wrong with Romania starting with the corruption, but I wouldn't say everything is shit.
Othwewise the local tech hubs wouldn't be full of westerners working here. I also miss the above average wages and the above average women. :)
Seeing the wages in Germany vs the property prices makes me feel poor.
Edit: Kostas announced on Twitter that Data Artisans recently also had an undisclosed Series B. So, the founders probably received a bit less that I wrote above. If the founders lost control of the board in the Series B, they may have received a good bit less than I wrote previously.
Once you are financially secure, living in Germany makes no sense. Shitty weather, the insane apartment situation in any major city, shitty services(any company with Deutsche in front - Deutsche Bahn, Deutsche Bank, Deutsche Telekom, Deutsche Wohnen), the beurocratic burden in your day to day live - everything takes forever to get done here. Spain is not perfect, but oh my god is Germany overrated
> Once you are financially secure, living in Germany makes no sense.
> How so?
A personal preference. The quality of life is not high, considering the bureaucratic hurdles you have to overcome on a daily basis - dealing with any Amt/Behörde(agency), Schools, Kindergarten, etc. And on top of that - you get penalized for working hard. There is this Goldschnitt situation(golden edge, breaking point), where the taxes get way too high and makes no sense to work above that level.
> They are going to have to be extraordinarily shitty to be worse than what I have already in the UK.
I spent a couple of years in the UK and the services in Germany are way worse. Banks are better in the UK, transportation is better in the UK, at least in major cities(options where you don't have to rely on your car), doing business is easier and cheaper in the UK. Maybe if we compare the NHS vs the german health system, Germany has a slight advantage, at least when it comes to waiting times.
> the insane apartment situation in any major city
Here again - bureaucracy and the car lobby, which gets in the way of any sort of public transport development/investment. So there is space, but people have to bundle around the well connected spots. For example: East Berlin is vast and empty, no transport connections developed for 30 years - only tram and the occasional buses.
> Shitty weather
> To me, constant heat and sunshine is shitty weather. I prefer a temperate/boreal climate. I find it far more pleasant.
It is 50/50 - I would not want to be in Spain in June, July, August and I don't want to be in Germany in January, February, March
It's not so much an active dislike as it is just not my cup of tea.
I've visited Spain a few times and it's nice in short bursts, but to live there I would have to deal with the weather (not everyone likes hot and sunny, I'd honestly prefer Norway's weather to Spain), the piss-poor developer salaries, the struggling economy, the food that I really am not a fan of, the poor quality of healthcare, the remnants of pro-Franco fascism in their political landscape etc.
It's just not for me. I think I'm more suited to Germanic/Nordic countries for culture.
I'm not citizen but working as a dev. I have an old passport which doesn't have bio-metric security features. However, I have a valid residence permit card (it already has bio-metric features). Revolut app which is UK based bank accepts my residence permit in identification process without checking my passport. Just scan it via the app itself and there you go. There is literally no bank in Germany (including N26) that accepts my residence permit card. I constantly have to fight with every company providing any online financial service, because their online video identification process (known as Post-Ident which is run by Deutsche Post) only scans bio-metric passports.
How come? If this thing (residence permit) is a valid ID document, then why is everyone asking for passport? My identity has already been approved by immigration office.
If residence permit is not a valid ID document, then why issue it at first place?
Slightly related is how it is mathematically impossible for VC to develop that "99 duds, one win" mindset that dominates SV when wins are not expected to become big enough to compensate that many losses. High failure rate VC cannot exist without robber-baron style exploitation of the occasional victory. Only bootstrappers are free to be modest.
The funding options in Europe are abysmal compared to US or even UK. EU response to this is various EU funding programs that are mostly throwing money away for little to no benefit. At least in the tech sector.
> EU response to this is various EU funding programs that are mostly throwing money away for little to no benefit.
You're framing this as if it's the EU's job. It isn't.
The only way the EU could compete if it would become a federal state, similar to the US. Right now the EU, despite being a single market in many respects is far from being a true digital single market.
Individual states are actively fighting against this because they would have give up bits of their sovereignty (controlling tax levels, such as VAT, commercial legislation, etc.).
You can't both have your cake (a huge single market able to funnel huge amounts of money into your desired investments) and eat it (keep total control of your country's finances).
> Right now the EU, despite being a single market in many respects is far from being a true digital single market
I think this is less about regulations and more about language. Most people in the EU simply don't speak english so you would have to have your marketing and sales in 20 languages which is simply not doable. A french startup just can't cold call a romanian company because they can not communicate. Yes, it gets better with each new generation but it will be decades before this is "solved".
How do these things get realized in the EU system ? I mean hypothetically if Germany, France, Netherlands, the Nordic countries pass legislations for greater integration wouldnt others follow suite ? Should it also not pass easily since most big businesses would probably get behind it ?
You're forgetting that Europeans are overall fiercely nationalistic by modern standards (not-WW1 nationalistic, but still).
Every EU country pushes back when there are attempts for more integration. More than that, there's a rising anti-EU current in most EU countries, due to various population movement issues.
EU capital markets are underdeveloped compared to those in the US. European banks are too small, and mostly national. More importantly, the marketplaces are too small and too national. European stock exchanges are a joke compared to NASDAQ and NYSE.
There's also the small detail of pension funds generally being forbidden from investing in stocks. That's the money that's fuelling SV, if my impression is correct. This policy comes from a general risk-aversion that's also evident in all other areas.
This article from The Economist touches on this topic.
“Banks cannot use deposits in one country to lend in another, because national regulators do not want to be on the hook for loans to improvident foreigners. An eu-wide deposit-guarantee scheme would allay that fear, but has yet to be agreed.”
I wonder what it would take to create the set of conditions for another "Silicon Valley / SF Bay Area" to appear in Europe. It seems like inevitably, any tech companies that are successful always end up getting bought out and leave to the U.S. instead of staying.
What are the primary issues? Lack of VC funding, culture, government policy? On the surface, it doesn't seem like there's anything particularly lacking, at least from my superficial understanding as an American viewing in from the outside.
> On the surface, it doesn't seem like there's anything particularly lacking, at least from my superficial understanding as an American viewing in from the outside.
My personal opinion is that, whilst we (still, just...) live in a "single market", Europe isn't a "market" just by itself.
Sure, you can do business cross-borders quite easily, but there's not the same critical mass of language, culture, business environment, and confidence as there is in the States. The US is a bunch of loosely connected areas that at least have language, currency, and to a lesser degree, culture that permeates them.
The differences between the UK and France when launching a product are substantial - and whilst it's technically "easy" (a bit of i18n here, a Stripe GBP-EUR transaction there), unpicking user needs and behaviours across language barriers can be hard.
If you incubate a product in SV, you can roll that out to the rest of the US with only slight changes in approach. If you incubate a product in Portugal, you have your work to cut out from the get-go to reach more than just 10m people.
My stretch is that it encourages companies to try and "dominate" their home market rather than trying to do the harder thing and spread - that slows the hockey-stick as you try and wring the last few dollars out of a mature environment and makes investment outcomes less tasty.
Combine all of that with a generalised point about Europeans being less "work-driven" and so probably a less % of people trying.
It's attitude and sheer luck. People in the US seem to take more risk, this leads to huge rewards (start-ups) but also leads to an insane amount of inequality in wealth. I have never seen more in-your-face wealth inequality than in America (and yes, I've been to China and Asia in general quite often). Then a second component is the sheer amount of people trying their luck. Imagine that winning in a startup is like tossing a coin with a p=1e-7 probability of landing on "win". The Netherlands has 2.000.000 people aged 20-30 and the US has 40.000.000 people aged 20-30. Imagine that all these people start a startup we have a probability of P(X>0) with n=2.000.000, p=1e-7 is given by 0.18 and the probability for P(X>0) with n=40.000.000, p=1e-7 is 0.98. The point I'm trying to make is that a lot or more people in the US try to start something, this leads to many more failures, but also to some singular huge successes.
Those 500 million are on average quite a bit poorer than the average American (regarding disposable income and therefore the willingness to risk a long period of 0 income to start a business). Especially in the Eastern part. Mississippi (!) has higher GDP per capita than most EU countries.
It’s not just funding - valley startups have a big group of experienced talent that knows how to scale in exactly the right moment. Startups and growth can die by small mistakes not making these small mistakes can compound quickly.
The other side is that the Valley is not a US place it’s a global tech place, the amounts of international money are staggering and additionally the number of international talent is staggering.
A third thing is that markets are not rational, if your investors are from the network they will hype you in that network, so your valuation is only partially dependent on product.
If you look at how China managed to build up an impressive tech sector the conclusion is clear: protectionism, else Europe will keep being sold off and lose its best talent to large companies abroad. This is the missing piece, not all these arbitrary number crunching or ill-informed cultural studies. Indeed looking at history many European countries have build up large innovative industrial and scientific sectors.
Naturally, if Europe starts to protect its tech sector the US will react aggressively and it will have major political repercussions. It is nonetheless necessary.
It's mostly generations of VC Funding. There's a genuine lack of experience on both sides so only the 0,01% of Entrepreneurs and VCs are capable of playing at an international level.
It's getting better, but not fast enough so that "just raise elsewhere" is a common answer, of course that's a route also most can't take for one reason or another.
I desperately want Europe to replace most services I'm using, because I prefer their regulatory environment. Primary examples would be gmail and google (I'm using qwant and it doesn't compare, but I use it before I switch to google). I do not use facebook anymore, and although I have a twitter account I do not use it.
Based on my daughters phone usage, I don't think that facebook is the thing anyways. It's all private chat rooms with her friends, which seems like a much healthier online interaction than what FB is. I just want to see that kind of thing primarily incubated and hosted in the European regulatory environment, because I don't trust the U.S. one where I live.
I must say I am quite disturbed by this mindset. Just imagine if European countries back then rejected Industrialization due to massive externalities (Pollution, child workforce, etc), they would not survive the Great War (WW1).
In addition, while big tech companies might be not very appealing in terms of security and privacy, you better have those which can be easily influentable and molded. That wont be possible without indigenous tech giants
The US lacks labor laws which means hiring people is a low risk activity since you can easily fire them. You can also work them for 60 hours a week and 52 weeks a year. The labor laws that do exist, for example in California, are about workers being able to switch companies versus them not losing their jobs.
That means you can quit your job, start a startup, hire people, shut down the startup and then find another job with ease. When you need to give two months notice to just leave a job that whole process slows down significantly.
China is definitely quite a competitive economy in the technology space. I wouldnt say its the next SV. Rather would group it as its own class. Most of the businesses grew out of protectionism offered by thr Chinese govt but noe they seem to have made it.
The protectionism was/is there in the US too. I unfortunately can't remember the details but there is a video in the Computer History Museum that tells a story about how computers from Germany were blocked in the US and the impact it had on both countries.
Of course, as the US now has a extraordinarily strong advantage in the sector it doesn't have to flex that muscle as often. But it does flex it when needed.
I suppose a look east policy will make sense here atleast for the VC funding. Language barrier problem is honestly an opportunith but also a stupid problem to have. I dont understand why atleast language of business is not English throughout the EU (for sectors besides IT). For example, quite a lot of businesses in China use English. Same is the case for most of south east Asia and also India.
I've been following data Artisans for a while and love their tech! It seemed that being not from silicon valley made it more difficult for them to gain traction.
One part of the acquisition is the integration of Alibaba's in-house modifications called Blink. It will be interesting to see what is behind that. To me it looks like this could be a very health collaboration.
It's an aqui-hire, i guess. Not great for Data Artisans customers. Alibaba are flexing their muscles. Google have their own data platform. We want to show we have our own too - they see it as competitive advantage. Alibaba are not an enterprise software company, like Data Artisans were.
An enterprise vendor would have been oracle or red hat who kept flink available and for sale on a clouds and on-premise. I can only buy assume flink will become an alibaba only managed service. Why else would they buy it?
They already are for American, and Chinese entering the market might actually introduce some fresh air to the stuffy atmosphere created by being at the absolute bottom of the American corporate outsourcing chain. Except in Germany and France - there they are cheap labour for domestic conglomerates.
