> 4-5 years ago there was a reasonable strong enthusiasm for Scala and functional programmers in general. No, I've been looking to change jobs for the past year and absolutely nothing has popped up.

I think businesses have weighted the power/capability of scala against the relative lack of easily hireable programmer for them.

If a business makes themself dependent on one (or a few) key scala programmers, they are more prone to be held hostage (e.g., the power dynamic is more towards the scala programmer). They will have to pay more salary or face the risk of the programmer leaving (which affects business continuity). The scala stack doesn't automatically give a more competitive edge against the competition, however (tho i would say it does, but only if the entire org buys into it).

Therefore, it makes more business sense, esp. for a middle manager responsible for hiring, to hire a java programmer (which is basically a dime a dozen at this point). The business will face no risk of the programmer holding the business hostage (because they are easily replaceable). So that's why you see businesses start to not adopt scala.

Why did they adopt Scala in the first place? I remember around 2012-2014 it was the hawtness. My guess is that there are languages that give you much of the power of Scala without the complexity - Kotlin, and modern Java, having caught up.

I worked at a Scala shop in 2011. It was pure hell. Slow compiles, bad tooling (the Scala plugin for Eclipse was a slow as molasses, IntelliJ's was a bit better though), inexperienced developers. I asked why they chose Scala... It was obviously immature at the time. "Joe thought it was good." (Joe was the architect.)

I'd never touch it again.

It's changed beyond recognition since 2015 let alone 2011. Seems odd to base your judgment of it on an experience 10 years ago.

My other experiences at that company definitely put things in a bad light. I should take another look at it, you're right...

> Why did they adopt Scala in the first place?

I think in the case of Scala many companies became invested in Spark and Scala came along for the ride. Then as time went on, it turned out you could do advanced FP in Scala, despite its warts, with principled libraries like Cats and the Typelevel stack (see https://typelevel.org/cats/). Companies could then use Scala as a "better Java" or squint-and-it's-almost-Haskell. Or like you said they could just ditch Scala and use something less powerful but easier to learn and work with.

Kotlin is going down the same path, with JetBrains trying to create an eco-system of its own.

I bet in a couple of years, as the Java VM platform keeps improving, it will only matter to those stuck in Android.

This is a movie with a script I have seen multiple times.

Modern Java is a language I'd be happy to program in, but doesn't have the language constructs you need if what you're loving FP in Scala (easily chainable monad-things). Kotlin is a nice improvement on modern Java, but feels closer to Java than Scala. Arrow is interesting in this space: https://arrow-kt.io/

Scala got huge because of Spark.

I'm a Scala programmer due to Spark. I love it.

Indeed. The only recent Scala job's I have seen in Australia have been Spark jobs because Spark uses Scala. Even these are diapering for Python Spark roles.

I've worked on a few big data projects for large companies. I have a huge dislike for them as mostly seem about how can we fuck over a customer to better the business or do shady stuff with the data rather than anything meaningful.

That and it's mostly just ETL, aggregating various data sources, cleaning, putting the data in some central location then running queries on it and the "If the only tool you have is a hammer, you will start treating all your problems like a nail." problem, tiny data sets going from NiFi / Hadoop / etc when `cat ... | awk ...` will be more than sufficient.

Yes for us Spark is being used for just ETL. It's internal data though. It's at the scale where no single machine could ever handle it though. It's replacing large proprietary systems like mainframes and teradata.

We wrote a framework to heavily simplify writing these ETL flows. No developer of the ETL flows is using Scala unless something custom is required. I wish we could opensource it.

>I've worked on a few big data projects for large companies. I have a huge dislike for them as mostly seem about how can we fuck over a customer to better the business or do shady stuff with the data rather than anything meaningful.

Sounds similar to the AI hype these days.

My theory is that Scala was adopted because Java stagnated. Java isn't stagnant anymore (even if its behind other languages) and a lot of people I know that advocated Scala are just fine with Java now.

I strongly identify with this. I lucked into learning pure, typed FP at work and I can't go back.

I think there are professional opportunities doing FP in Haskell/functional Scala, though they are few and far between and in certain industries such as fintech or other areas where correctness matters more. Your best bet is to look for a remote position and keep your eyes peeled (at least, that's my strategy!).

As someone who works for one of the companies in Sydney that happened to use Clojure (one of the reasons I joined them), I know find myself in a strange situation where as a hiring manager and a leader of team of engineers that have little or no interest in FP languages that I can't justify advocating for FP languages.

a) hiring is way too difficult at the best of times and is compounded more when you filter the pool so small (admittedly the filter is not bad for some measures).

b) so few at the company have any interest in using languages outside of the more mainstream; for example it was no effort at all to get interest in Golang.