Python 2 advocates were quite firm in telling us that Python 3 and its incompatibility with Python 2 had killed Python and people would abandon Python altogether in droves for other languages.
This post is suggesting the opposite, that Python is more healthy than ever and growing incredibly fast.
Three of us (including me), in an academic lab, are making the jump from matlab to python. We use py3 (of Anaconda, on Windows10).
We like it, but we are constantly screaming and bitching about the complications of 2<->3 ("Ahhh X and Y module only works on py2!", "Ahhh device developer only provides example code for Labview and Matlab!", "Ahhh python isn't even supported for this device/application!").
Package management ("Do I use conda? pip? easy_install? compile?") is annoying, I hope that something like hatch (https://github.com/ofek/hatch) maybe takes over everything -- because I think the mess that is package management is THE biggest deterrent for newcomers.
I think the one thing we are all learning from this experience is: JUST USE UBUNTU, doing python with Windows is a headache.
All of that said -- python is nice, it's amazing, it's the newer, funner Java. It is incredibly approachable, it can do pretty much EVERYTHING because of the expansive availability of libraries, it's finally a programming language that is great to get my 7 year old niece started on.
Package management was so frustrating coming from Ruby to Python. There are so many programs, and version conflicts that I run into all the time. With gems and bundler, it was rare if things didn't work.
If you get into the habit of making a virtual environment and requirements.txt in every repo then it's smooth. Just activate the env when you're using the repo and use vanilla pip install to install dependencies. Then pip freeze them into the requirements file. It's a very similar workflow to a gemfile and bundler.
I worked on python for a few months last year, and although I didn't find anything representing a canonical description of that workflow, I did eventually come to this conclusion. The only thing was that pip freeze > requirements.txt didn't really give the same power as I was used to with Gemfile.lock and the various permutations of bundle update. I forget the specifics, but I remember being unable to get the finer points of optimistic version locking to work in a way I found acceptable.
That said, I have been doing ruby since before Bundler, and I really have to take my hat off to what Yehuda and company accomplished with Bundler. It was both a technical and open source community triumph to get Bundler done, stable and covering the breadth of use cases it applies to.
You're going to have a hard time supporting multiple versions of Python that way. If I started a new package today I would target 3.4, 3.5, 3.6, 3.7-dev, and likely, 2.7. A lot of well maintained packages have different dependencies based on the exact version of Python. If you're crazy enough to support 2.6, then there will be a lot of additional packages in pip freeze.
Ideally, packages would have just one set of dependencies and the packages would be version locked, but that's just not the case in the Python community.
They plan to release the spec like that, but I'm certain on this update most studios will be lagging due to the vast amount of 2 to 3 migration work that needs to be done.
But be sure to use Pip into this year's flavor of virtual environments rather than globally, since you might have multiple projects that need different versions of the same package. And be sure to run the right version, I've seen situations where a computer had python 2 and 3 installed, somehow python3 became the default when running "python" but if you ran "pip" you'd get dumped into the old version.
And if your editor has a "Run script" function, it also needs to deal with "Activate the correct environment" for whichever flavor of virtual environment you've used.
Any what beginner would be confused by tutorials using all sorts of different setups (virtualenv, venv, pyenv, pipenv, virtualenvwrapper, pyenv-virtualenvwrapper)? Most of them, I think.
"venv" is the newest one - shipped in the python3 package except when it's not (Ubuntu, maybe others) and doesn't make copies of the python binaries for each environment. That always seemed wasteful to me, but I suppose it was the easy way around path problems with the ecosystem not having a solution built in.
I love python when I'm only using the standard libraries, but if I have dependencies it's messier to set up than I want it to be.
Honestly, virtualenv works. Every single time. I use virtualenvwrapper because I find it and the aliases it provides useful.
I'll stop using virutalenv when it actually becomes a hindrance. I know venv exists, but I have precisely zero incentive to use it when virtualenv works and allows me to get on with what I actually want to do.
virtualenv is python2, venv is python3. virtualenvwrapper is as described, a wrapper around them to provide some convenience functions, but they're perfectly functional without it.
Those all seem like pretty basic developer best practices to me.
Of course each project needs it's own virtualenv. In fact, each project should have it's own docker image. Why install an application library to the entire system? That is not a good idea in ANY language.
And of course scripts need to use the right virtualenv. Or better yet - why not put each app in it's own docker? That bundles up everything into a single "file" you can run on command.
All of these things you are listing as downsides seem like things that one normally moves away from, when they put a codebase in git and stop deploying to prod by copying a few files to a shared network drive and restarting the server?????
> Those all seem like pretty basic developer best practices to me.
Those are all bleeding edge developer best practices that didn't exist five years ago. Folks coming from matlab (an example I read above,) probably aren't in the industry and many likely don't even know docker exists OR why they would want to use it.
And the GP was saying that the ruby package manager is more magical then the python package manager... when pip solved all of the complaints mentioned.
I do mostly Ruby professionally and dabble in Python, and I agree with heydonovan. Pip works fine for installing things, but near as I can tell, Python has no equivalent of bundler - a tool to install specific versions of a bunch of libraries, generate a lockfile ensuring that your production server and all of the devs are all running it with the same version of everything, and letting you have as many versions of a gem installed as you want, and still running a project with specific versions other than the latest.
You can do this, but it's not as easy. The pipfile project is an effort to improve this.
The current standard way to do this is with a virtualenv and a requirements.txt file in which you specify the version of your dependencies, which you can then install with pip install -r requirements.txt.
Format is "packagename==0.1.0" (>=, < etc work too). VCS urls work too btw, e.g. "git+protocol://site/repo@tag_or_commit"
To get something similar (but inferior) to lockfiles you can use "pip freeze > filename"
That's not true. A requirements file is the canonical way to do this. Just use a requirements file and pip install -r requirements.txt. Then version control that file and other Devs do the same. Just like a gemfile.
> Python has no equivalent of bundler - a tool to install specific versions of a bunch of libraries
Zope and plone, the project(s) that came up with python eggs, used to have the concept of "known good sets"[1], which were massaged and used with zc.buildout :
I'm not sure I'd recommend it for greenfield projects - splitting things up in sane-sized chunks, and using the now in-standard-lib vevn (python3 -m venv) is probably a better idea.
But just for the record, there exists a canonical(ish) solution to some of these issues in python-land.
Sorry to nitpick, but Zope and Plone were not the originators of Python eggs, although they may have popularized them.
The design was done by Phillip Eby, with funding from OSAF for the development of Chandler, not Zope.
Much shade has been thrown at eggs over the years, and people bemoan the warts, but eggs and setuptools were better than what we had at the time in Python.
requirements.txt don't track transitive dependencies. Your project could break if a dependency decided to swap out its own dependencies. This isn't theoretical either - it happened to us in production.
This is only practical if you only have one set of requirements.
If you have multiple sets of requirements (for dev vs testing vs production, eg), this becomes impractical real quick. You'll essentially have to tear down and rebuild your dependencies multiple times to update your reqs.
Things are a lot better with Python 3 on Windows, and especially so with Python 3.5+.
"pip install numpy" will actually work, among other things. But even for packages that don't have binary wheels and have to compile things, it's much easier now to install a compiler toolchain (it's actually a single download!), and have your Python installation just pick it up automatically.
> Things are a lot better with Python 3 on Windows, and especially so with Python 3.5+.
For one of my main applications py3 proved to be too slow. For my application I use: numpy, pil, tkinter, and subprocess (and process/produce/draw on screen images (at a rate of 120/sec)). I don't know WHY but with python2 it runs acceptably fast, on python3 it's 20% slower. This has been something common for me actually.
I do tend to use py3 when doing any kind of data science though.
If you're using Windows, then WinPython (http://winpython.sourceforge.net/) is an absolute necessity. I have two separate installations, one for 2.x and one for 3.x, and I have zero issues. Works amazingly well and comes with just about everything you need already installed and working. Can't say enough good things about it.
But Java was an overnight success, while it took Python years to get traction. I was exposed to both around the same time, probably 1995, but I ignored Python until the second time I encountered it in 2003. It just didn't have the hype.
Java was by no means an overnight success. It was way too slow for several years until they really started to do anything more than run the bytecode in the JVM.
"we were able to hook Java runtime onto the free, open source Netscape browser. And it was a collision of two things that accelerated and launched each other together. It was fascinatingly lucky timing for both companies."
While it may have taken time to mature at that time it was an "overnight success".
Sun was considering dropping it and considered it an expense before that.
I'm curious, what issues do you run into with python3 incompatibility?
Most scientific stuff I've done works in python3. In fact, its now possible to run a tensorflow/opencv/scipy environment entirely from pip in a virtualenv, though it won't be the fastest. which is amazing. Doing that kind of thing in any other language would require docker.
In general, if you're sticking to python3, the answer will always be "use pip", everything should work with pip, and everything should install as a binary without needing to build source.
Edit:
There's exactly one context where I use python2, and its for a robotics project where some transitive dependencies are python2 only.
I think GP is talking about virtualenv, not pip. Using an optimized build in a virtualenv environment should not be any slower than using it outside the environment, right?
Actually I always complain about Python packaging when I'm only using Python. But the moment I start using it in other languages I dream about the comfort and professionalism of tools in Python.
Of course in scientific Py you need compiled C attachments, which makes it more complex. But in normal Py usage you only need pip. Easy Install is only needed when you don't have pip installed. Pip is a drop in replacement with additional features.
And at least from what I heard for everything with C compiled stuff inside ("binary packages") you use conda.
Well, as said before sometimes you need it because pip isn't installed by default.
Just had such a situation yesterday at work. A guy from QA (not a dev) needs to setup an environment to install our tool for testing. He doesn't know python, so explaining to him what a virtualenv is would take way too much time. He probably also doesn't know how to use the OS's package manager to install pip. So, what to do? Give him the oneliner "$ sudo easy_install whatever". It works, he doesn't need to understand it, and actual system maintenance doesn't matter because it's just a VM that gets deleted after he's done anyways.
In fact, it's always a _SURPRISE_ when I see Python being supported. I would say after having gone through about a hundred major lab devices, less than 2% I found to be supporting Python.
Importantly, example code should be made available (as it generally is for matlab, and labview) because creating things from scratch in a move-fast environment is a little too much.
I think you're right, the biggest pain for a general developer for Python is determining which package manager to use. I'm thankfully insulated from it a bit due to using Docker for a lot of my development, but I hate the fact that Pip won't install to your home/project directory unless you coax it to (and sometimes still won't even if you give it all the proper environment variables), and I think virtual environments are not the solution (I shouldn't have to think about how to install a package or work on a project). Especially coming from Node-land, and before that Ruby-land, Python seemed like a wonderfully designed language but with terrible package managers. From what I remember Pip is going to change the default to install on the home directory sometime in the future, but it shouldn't have taken years of discussion to come to the conclusion that it's the right thing to do.
I don't really like virtualenv because you have to activate it to do anything and deactivate it if you want to go to another project or work on something unrelated. I'd much rather it work by the project directory so I don't have to worry about the activation part (this is why Node's package manager, for all its issues, is great: you install in a local folder unless you explicitly supply the -g flag). At least I should be able to install packages to my home directory.
I still think that the "project" directory is best, but I'm glad that Pip is moving to local-by-default in any case. It won't take away from the necessity of virtualenv for cases of different dependency versions, e.g. Django 1.4 and 1.10, I think, unfortunately, but it's definitely better than global by default.
The problem is not for people like you who already decided, the problem is for beginners who aren't aware of some of the 'envs' and are looking for recommended practices.
the #1 issue i have with py on win is having to download a million different C++ compilers esp for packages not distributed on pip, and esp the more obscure ones with no support whatsoever (e.g. data processing or math packages that were put together by some chap who was working on it for a specific set of research papers and has since moved on).
chris gohlke's website was a real lifesaver back then, although now pip seems much better. nowadays it can be a bit difficult replicating environments across different machines especially since conda is moving rapidly from python ver to ver while some obscure packages i rely on are still stuck on 3.4 or earlier... or if c++ 10.0 or 9.0 or 14.0 is not set up correctly. then it becomes an issue of crossing fingers and hoping "python setup.py install" doesn't fail.
What's the point of using Windows as a Dev or data science environment anyway? Everything is built and works well in Linux, scripting is powerful, it's free, etc.. seriously asking.
When you work for a company that gives you Windows PC's, and have a horrible Linux environment (through a network, not on your laptop). From a Python standpoint, there is no down side to developing in Windows.
It's easier for me to install the Python libraries I need in Windows than to install them on Linux where I don't have root privileges.
I really don't know where this is coming from. The only people I hear talking about a 2v3 divide are my Javascript developer friends, who are saying that as outside observers.
As a Python dev, I (and all the other Python devs I know) were just waiting for (and helping) libraries to start working with 3 before we can switch over. These days, all my new projects are in 3, 2 is pretty much for legacy code only.
Same experience here -- got back into dev earlier this year and decided to go with Python 3 everywhere. The only libraries I've needed that were Python 2 only were ones that weren't being maintained anyways. It's been a pretty good recency filter for me. I know that wasn't the case when Python 3 was younger, but the ecosystem is pretty dang good today.
Those people were advocating for stability, not necessarily for P2. The 2->3 was a pretty big screw up, and it definitely had an impact on the language and ecosystem.
Python seems to have overcome that (keyword: seems), thanks to a strong offering in data science, but 2->3 is still a headache. I never really used Python for application development, but rather for utility/scripts and I am still on 2, because I don't need more.
2 is available out of the box on all OSes and at work we also use 2 for building tools.
I have the feeling that Python is still losing people to Golang due to its weak performance and Golang's reputation in that area.
So instead of being arrogant about 2->3, I'd say Python devs should be thankful to data science for saving their butts. :)
That's not the way to look at this. The control you would need to establish whether what you are saying is true is a world in which the 2/3 split never happened. And in that world Python is most likely doing even better than in this one...
They fucked up. That Python is resilient enough to absorb such a blow and that no other language is a viable alternative for that particular niche is what saved the day.
And even now there is still plenty to improve on, fortunately Anaconda at least makes package management more sane.
Why are you deliberately making things harder for yourself? You're going to need to upgrade these projects within three years. Just seems like a wasted opportunity not to get off on a strong footing.
My Python 2 code already converts cleanly to Py3 with 2to3. There will be zero effort for me to port anything when 2.7 is EOLed. I just want to be able to use the new features without dancing around the forward compatibility game.
I feel like it's still a better idea to migrate back to python 2.7 and update it to match python 3 features in a backwards compatible way. At the very least there needs to be a way to interop between python 2 and python 3 code without modifying the python 2 code.
I haven't used python for a while though. Maybe they found a solution to the python 2/3 problem already.
As someone who works with Python every day, this isn't surprising. I forget where I heard/read it, but someone made the claim that Python is the "second best language for everything" (paraphrasing) and I couldn't agree more. Need to create a web app that interfaces with a data pipeline? Python/Django/Flask is perfect. There are countless other examples too.
Why second best? I think it is the best for almost everything. One reason is that its grammar is so intuitive, that you only need to think about the problem to solve. With other languages, you always need to look out for grammar issues.
auth is a must-have (i've never written an app that did not need basic auth) and admin is a very, very nice to have. It still feels "thick" to me in other areas.
Agreed! I lean much more towards Flask (I'd rather have too much simplicity than too much complexity), but something in the middle would be fantastic.
A simple application structure with loose enough rules and a strong standard library is what I would like. Flask can be too little, Django can be too big.
I believe the idea is that the best language for any given situation will vary depending on the situation.
The point, I guess, is that Python isn't really optimal at anything but is very good at a lot of things.
Realistically, there are enough software fields that Python just isn't applicable for this to be taken with a pinch of salt. However, as a language, it does have quite a large sweet spot: easy to pick up, expressive, integrates well with low level languages for heavy lifting and almost perl quantities of libraries.
IMO its popularity is justified although I'm not a fan myself.
It depends on the context. The joke is that everything has a most optimal language and Python is never the most optimal language- but it has got breadth.
This is probably related to explosion of interest in AI and deep learning (DL), because Python has become THE standard language for AI/DL research.
Most AI frameworks support Python out of the box, including TensorFlow, PyTorch, Theano, and CAFFE/CAFFE2, which together currently have the most developer mindshare. Every single TensorFlow, Pytorch, Theano, and CAFFE/CAFFE2 tutorial or example is written in Python.
With very few exceptions (e.g., the occasional Lua code coming out of Yann Le Cun's group at FB), new AI papers come with code written in Python.
If you want to see Python's utter dominance in this field, just search for "deep learning" or "neural net" on github.
To be fair, this was also true when I was using Matlab back before switching to Python (2006ish!), except you had to dig through documentation instead of just having everything immediately available on SO. :)
the matlab documentation, imo, is the best technical documentation ever written for anything. if your background was more math-heavy then the docs provided easy to understand translation between recognizable mathematical idioms into code (if matlab wasn't already very math-idiomatic).
and back in 2010 or so, python's numerical optimization libraries left a lot to be desired, and totally hopeless compared to matlab. (edit, i don't remember it being nearly as good as octave, which wasn't particularly good either back then).
hmm... no? I think you can hit ctrl-d or some shortcut like that to go straight into the code of the function that you are interested in, and it can be functions written by matlab.
Predicting it now. A large part of the "why" is going to be because of the jump in data science. Python is an incredibly kind language to new comers in the sense that it allows you to just get started. A lot of people are making bigger jumps into data science (purely anecdotal) in the sense that they are programming more around it. Scraping data, cleaning data, etc is also really easy with Python.
Devops stuff is also a great one to pair with Python.
Reason I'm predicting this: I've kept an eye on Python+Flask /Django jobs for over a year now, and while I see "Python" show up more frequently now, "Flask" and "Django" are still about as frequent as they were in the past.
Looking forward to the "why" post. Glad that they mentioned that they'll be publishing that soon.
One "non-programmer" perspective here: Python is the new Perl, except it's fast to read AND write. In Python it's fast to do ...everything. I'm an electrical engineer by trade, and Python gets used a lot.
* If I have a giant CSV with data in it:
import csv
with open(filename) as csvfile:
csvreader = csv.DictReader(csvfile)
data = list(csvreader)
Bam, an array of dictionaries with the data in them, indexed by column name. With listcomps, whatever I need to do will only take a few more lines. And it's this easy for almost any data format. There will be a widely used library for me to `pip install`.
* I needed to plot some I/Q data and superimpose some other data as a heat map over it. Complex numbers are built in. Matplotlib, and bam, graph created. It looks as nice as the graphs my coworkers make using Matlab, and I coded it up just as fast, if not faster.
* I wanted to play around with ordering a pizza online automatically. Import requests, and off I go.
* The code is so easy to read, I've had coworkers update things I wrote when they have never programmed in Python before.
I don't have to do any compile steps. I don't have to use a special IDE. I don't have to think in objects and state-passing and all that BS (but I can if I want to!). I don't have to search for some arcane library. I don't have to know a magic build sequence. I don't have to edit config files. I just make cool shit really fast. It runs slower, but for most things I use it for, I don't care if it takes 10 us or 10 seconds. I just want to make it work.
I work at Karolinska as the sole developer on a data browsing platform, inside a neurobiology research group. The entire department is either using or learning Python. Of course, the work environment itself is a severe form of selection bias, but there definitely has been an uptake in Python among scientists.
(Which leaves me kinda sad, because I like Julia more - I'm sure it will keep growing though)
Julia is definitely a step up in data science programming. It's purpose built for data science. I guess that's also the reason why Python has grown more than Julia. The likelihood of someone having worked with Python in an unrelated field means that when starting out in another field, if the Python option exists, they are probably going to take it.
Apart from that, if one is just getting started out in data science and has never really handled a programming language before, Julia is very nice really. And their documentation is pretty damn awesome. And once you get past the performance recommendations I think it gets pretty fast (?). I'm out of my depth beyond a certain point since I've only ever experimented with it.
The problem is that data science, often, isn't just data science. It might involve doing a bunch of activities that are strictly not data science - scraping data off the web (Python has requests), visualization for some understanding (Python has dozens of nifty libraries), store data in a small db (Python works nicely with sqllite, among others), convert all of this into an app (Python has Flask, again among others).
R's data frame concept was fantastic for those transitioning from 'spreadsheet' analysis but with Pandas, Python has improved and enhanced this concept even further.
This is just one example but I think it is this versatility that makes it more attractive than R or Julia.
Julia is really fast but the DataFrames/DataTables package was kind of slow and feature-incomplete. Because Pandas is so optimized, Julia wasn't winning out over Python on the speed side of things, and was definitely losing on the ergonimics, so for data science it wasn't the best choice (unlike say numerical simulation).
I'm sure this will change if it hasn't already, especially with the better parallelization that Julia is capable of.
> Looking forward to the "why" post. Glad that they mentioned that they'll be publishing that soon.
I think the "why" is because, for many users:
* they want a small and easy, dynamically-typed, garbage-collected language with few surprises (see `import this`). Perl misses out here.
* they want something built with C so there's easy access to native libraries. Languages on general-purpose VMs miss out here.
* I don't think they want something that transpiles/compiles down to some other lower-level language (which I suspect leads to difficult debugging). Maybe Nim misses out here.
* they're comfortable with imperative-style programming. Maybe lisps and Haskell miss out here.
* Python is fast enough for most things where C-speed isn't needed. Very-high-level languages chasing dreams of high performance mostly miss out here, IMO.
* they want a language that helps them catch their mistakes. IMO, maybe Lua misses out here (with default nil everywhere).
* Users want a general-purpose language. Again, Lua misses here, though, of course, Lua's goal isn't to be general-purpose anyway.
* I think many users also want a language that's licensed GPL-compatible. Clojure misses out here.
* they want a nice helpful community
* Users want a language with familiar C/C++/JS/Java -ish syntax. Python misses out here! {ducks}
Python has its own set of problems, but it hits all those marks except for the last one. So, it's gotten quite popular.
> It has to be about the most readable of all computer languages.
Quite readable, sure. Though my point is that curlies and semicolons are more familiar to many (and easier to edit/navigate in my editor, for that matter), and probably would be preferred. My comment was about why I think many (not all by any stretch, of course) choose a language like Python.
Thats a pretty awesome list!. I'd love for these to be the reasons but somehow I doubt any of us are that thorough.
The only reason I used python is that it was the first result that showed up in a Google search "write program to scrape web page" (ah beautiful soup. How I love thee!)
The only reason I used flask was Udacitys course taught their web app course with it.
I'll admit it, I've never actually analysed the benefits of a language in that way. My only judgement has been "mmm does it feel productive to use this language for what I'm doing". And "how quickly does the documentation take me to a hello world". Which is terribly subjective I admit. I'm suspecting I'm not alone. Or at least I hope I'm not :D. Yes. The embarrassment I feel is real.
PS - it's also how I ended up using nginx over Apache. I couldn't really understand Apache in 30 minutes so I used nginx instead. I even remember an interview where I was asked why I used nginx over Apache and I said "um... because it was easier I guess". Definitely not best practices and I'm trying to be more objective nowadays.
> I'd love for these to be the reasons but somehow I doubt any of us are that thorough.
Hm. I think there's a difference between reasons for trying out a language for some random task versus staying with it. I've bounced around between lots of different languages, and I suppose the list I presented above are why lots of people stay with Python.
I think you're judgement of "mmm does it feel productive" roughly corresponds somewhat with the above list. Though I accidentally left off "good-enough docs", which I think is a given.
> I couldn't really understand Apache in 30 minutes so I used nginx instead.
Yeah. Though I think it has a lot to do with how good the docs are. Definitely, if a project can't tell me what it is, why I'd want to use it, and an overview of how it fits in, I'm very hesitant to pursue it further.
I actually keep a jupyter qtconsole open to use them for ad-hoc data visualization. Pandas replaced excel for me a while ago and I cringe every time I need to abandon seaborn for a Tableau workbook these days.
GUI Visualization tools like Tableau or PowerBI seem to error towards presentation, while the defaults for seaborn help discover and visualize data while still producing results good enough to make C*O's happy.
Three years ago, I was shutting down my startup which was a webapp built in PHP using my own framework. I decided I was going to consult and I also decided it was time to learn a professional, modern framework.
At the time, RoR was hot and I was leaning towards it. Laravel felt good too because it was new and it seemed to do PHP web "right". The last option was Python and Django. What I liked about the latter is that data science seemed (still is?) the lingua franca of data science and I figured I could learn web programming and build up my Python skills that I could one day apply towards data science.
I ultimately went with Python and haven't looked back. Django (particularly DRF - shout out to Tom Christie if he's reading) has been an utter joy to work with. Even though we're in the era of fat front-ends, it's nice to have a bullet-proof, well supported backend.
As someone who has been using Python since 2000 roughly, another minor factor (besides the obvious data science factors) might also be the decline in Ruby. Ruby (and Rails) growth in the 2005-2010 era, seemed to noticeably put the brakes on Python's previous growth up till that point. Back then Ruby had all the boosterism/hype and Python was the supposedly "boring/legacy" option. Even now, Python's growth doesn't seem driven by hype the same way Ruby's was. It always seemed to be a more pragmatic choice rather than a fashionable one.
Although somewhat ironically, I don't use much Python any more now that I work at a mostly Ruby/Rails company haha.
I don't think hype is the main factor. Ruby's decline is due to its narrow focus on web development and Rails in particular. Whilst there certainly are gems covering other areas of computing Ruby never seemed to diversify the way Python has. With Node.js eating Ruby's lunch in the web development sphere Ruby is left looking like the single-commodity Venezuelan economy. Let me add that I think Rails is a fantastic piece of software. Maybe TruffleRuby and Substrate VM will revive Ruby's fortunes. Matz's promises for Ruby 3x3 are too little too late, I fear.
Having done both Ruby and Python, I confirm that they are pretty similar languages, both performance wide and syntax wise.
Python is mainly popular because of historical reasons. Not because of some magical functionality it has over ruby, here are a few key points:
Ruby was created in 1996 and Python in 1989. Python was the main language chosen by academia because at the time no one really knew about ruby, until its later explosion in popularity thanks to Rails in 2000's.
It was initially used at google which gave it a huge boost
It is also the go to language in the scientific community.
Ruby_can_ become a perfect candidate along side python should there ever be competitive enough scientific libraries just like NumPy, SciPy and so on.
That being said, I love both languages and communities!
Ruby suffers from indifferent stewardship. Yes Ruby could have competed with Python in the scientific sphere but it didn't and now it's too late. I actually think Ruby is a better language than Python, by the way.
It is not just Matz but the community or core team as a whole. Look at every Python 3.x release, and then look at the release in Ruby, some of the bigger improvement were more like catch up. And since then there is little changes.
It is also rather unfortunate no company wants to invest in Ruby.
Yes, but if you look at penetration testing and hacking tools apart from Metasploit its all Python. Again, Ruby had something good - Chef and Puppet - but failed to capitalise on and extend its domain. Now every kid new to hacking thinks it's synonymous with Python.
We've standardised on Python and we're migrating all of our systems to Python / Django / Postgres from PHP / Drupal / WordPress / MySQL. We started out with one senior Dev on the Python stack, and simply found that the productivity on Python/Django just too compelling to stay on PHP. The (senior) Dev on Python had a couple of highly successful projects which resulted with him getting some junior Dev support. Then the other Devs started to notice and saw/felt that they were missing out on better and more advanced tooling. They then started to push to move their main environments to Python/Django. This has further been pushed by the business stakeholders who have seen the Python/Django projects move much faster than PHP projects.
So for us, that is now 6 Devs on Python/Django plus some consultancy support. So we've move from 2 python Devs to 6 Python devs within the last year.
While PHP has gotten better recently, the language is not nearly as powerful or complete as almost all of the popular alternatives. Up until very recently, something like the Django ORM was impossible to express in PHP. Take an honest look at the available frameworks (especially in comparison to something archaic like Drupal) and you'll find there's no competition.
> While PHP has gotten better recently, the language is not nearly as powerful or complete as almost all of the popular alternatives.
I can prove that python and php are equal in their "power" because I can build compilers/interpreters of one with the other.
If you are arguing that python has better libraries then it's possible though for web development, I'd say most popular languages have similar libraries and capabilities.
But my point was more to the point of expertise in stack. A skilled PHP developer is going to be just as productive with the PHP stack as a skilled python developer.
People who think one stack is better than the other usually base their opinion on poor/bad developers.
There are plenty of horrible python developers as there are php developers.
The "power" of most stacks, especially something closely related as the PHP and Python stacks, are similar.
The difference in productive boils down to competence of developers, not the stack itself.
Python certainly has a lot more software written for it. I use it for scientific computing.
For building websites, php has plenty of good options too. I do really like the Silex framework for website construction. I backend process with java or python when needed to do any substantial computations.
The "standardization" on common libraries from symfony and zend, have made everything a lot better.
Referencing a micro-framework shows just how out of the loop you are. A better example of a PHP framework is Yii, but you still see tons of ambiguous arrays used because of the lack of expressiveness in the language.
Zend is not even a contender for modern web application development. And again, the gratuitous use of arrays for configuration highlight the shortcomings of PHP.
And mentioning Yii here instead of, say, Laravel or Symfony (though I loathe the latter) makes me wonder how in the loop you are, yourself.
I've not been a PHP developer for years. I wouldn't go back to it. But pretending it can't compete with Django is one of the weirder things I've seen on HN in a while.
Django everyday since 0.9.6 Still an active maintainer of a small PHP lib, though.
> I wouldn't go back to it.
That's what I'm expressing: there's no reason to choose PHP in 2017. This isn't misdirected 'PHP hate', but a genuine disagreement with the OP's statement:
> There is nothing inherently "more productive" on one stack over the next.
> There is nothing inherently "more productive" on one stack over the next. It's just technologies which are fundamentally similar.
It's hard to quantify the differences, but obviously something like Brainfuck is going to be inherently less productive as long as we're talking about human productivity.
Does the usage of a site like SO (where you ask for help) necessarily correlate with growth of a language? Or more specifically, does the difference between views of two different languages on SO correlate with real world difference of growth between those two languages?
As a personal example, I've searched for answers related to golang far less than I have for python because I tend to find my answers by simply reading golang docs but in case of python, docs do not help me as much so I turn to google and consequently to SO.
Not the usage in absolute terms (as it may just indicate the language requires a lot of SO research to do anything useful), but the growth of such usage would seem to correlate with growth of the language.
I moved from my last job (mostly C#/vba, front office finance)to a new role (python, data guy at hedge fund).
Doing data stuff and simple web services is absurdly more straight forward in python, the main things I miss (weaknesses of python to C# and I'm guessing java):
nice parallel options (I know several options exist but haven't found any of them as easy to get into as C# async/await, GIL is the problem i guess)
the django database layer doesn't do smart diffs in the same way as .net db projects (in .net it's smart enough to actually look at your code schema vs the database and work out how to roll forward/back, in django it's just using a combination of your code schema and a table describing what has and hasn't been rolled out yet, making everything a bit scarier and tougher if anything goes wrong. I dunno if SQLAlchemy does this.
edit: also I once read a HN comment that a problem at the heart of python is that block-syntax forces you into having only trivial inline lambdas, after writing it for a while I think they might have been right
It's just too easy. I never programmed a thing until after I had graduated college and yet when I did start running into times where I wanted a little script or program, it was always less than 100 lines in python to set it up.
TL;DR: Slightly interesting but narrowly focused article with clickbait-ish headline. It charts the growth of Stack Overflow python question views in the last five years. While this is an interesting statistic there is no additional analysis that it correlates to python's usage growth. And no hypothesis is given for the growth in question views, although they tease a future post about that. It would have been far more interesting if they correlated their stats to lines of code in github or something along those lines.
To me, if anything I'm curious why python was so low percentage-wise in question views only five years ago. Anecdotally it doesn't seem to me that python usage in "industry" has grown that dramatically in the last five years, but there could be other factors involved such as education. Or my biased gut instinct could be completely off.
I think even though Python in the industry like in web isn't growing as much as say Node, in other places like Data Science and Intro to programming in Unis and Schools Python is growing very fast.
The article is not narrow minded but has a narrow focus and only analyses concrete data from a single source. It may be globally accurate but indicates a rising trend in people being interested in python. While Python usage cannot be directly correlatd to SO, we can infer that languages that have more questions have more people using or learning it generally. In the same way languages with a small community of users don't always prefer SO but something more personal like Mailing lists or IRC (eg. J language).
It should be safe to conclude that due to Data Science being the new buzz word and python at the forefront of it all, atleast python is getting a lot of attention even if it doesn't translate directly to more users right away.
I just edited my comment to say narrowly focused rather than narrow-minded. That is definitely what I meant to convey but chose the wrong words, thanks for pointing it out.
> To me, if anything I'm curious why python was so low percentage-wise in question views only five years ago.
The majority of the top CS colleges switched to teaching using Python within the last five years. This provides a big reason for companies to use Python as well, since it drives down the cost of hiring.
As others have pointed out, more people coming out of school know it.
When I started my job, the company (fairly large one) was mostly Perl. Perl was the big language of a past generation, so when those engineers were hired, Perl was king. I was told by at least two people that having Python on my resume is not a boost for engineering companies (not counting software), as most managers had never heard of it.
Not too many years later, a lot of new work in the company is being done in Python. Why? Because that's what the newer employees know and prefer.
I think in the software world, Python was big a long time ago (even in the days I joined). But non-software companies were stuck with Tcl and Perl for much longer. Now that they're shifting, you will see some of that growth.
I get this same thing from the Delphi users forum, where the diehards strain to convince themselves that Delphi has "three million users" and "is about as popular as Python". They insist that there's no reason Stack Overflow questions should be correlated with real-world usage. One suggested, seriously, that perhaps Delphi is just so easy to use that users don't need to ask questions. This excuse has been adopted to explain a lack of commercially published books ("maybe everything to say about Delphi has already been said") and there was even an attempt to explain away job posting data. In that case it was theorized that Delphi jobs are filled as soon as they're posted while Java jobs go unfilled for months so when you see thousands of Java jobs on Dice but no Delphi jobs that actually means there's a huge Delphi jobs market and no one's actually using Java. :-(
Never underestimate how far users of "sunset technologies" will go to convince themselves that their skill set is outdated. Heck, Embarcadero's C++ product manager insists that "customers tell him" that Delphi (Pascal!) is "five times more productive than Python" and he actually believes this too. The Delphi product manager, meanwhile, told me he sincerely believes that Delphi has had more of an impact on the business world than Python ever has. Meanwhile, the former VP of Developer Relations once polled the first 500 people to upgrade to their newest release of Delphi, crunched the data, and concluded that users really love Delphi (of course, the many who felt the upgrade was light on features and high on price and chose not to upgrade weren't reflected in the data).
Did you actually read the article? Of course generating more questions is likely indicative of more usage. But the graph shows it from being in last place (of the listed languages) in 2012 with ~4% of the question views to being first place right now over all other languages with ~10% of question views. Python has grown a lot over the past 5 and especially 10 years in wide usage but I've seen no other evidence that the chart in the article should be believed to have a direct correlation with usage. There are many possible explanations of a more indirect correlation that could be skewing the numbers in unexpected ways but none are discussed in the article.
I'm the biggest python fan you'll meet (ok I'm sure that's not strictly true) but I followed the headline hoping to see more concrete data to support the claim of "Incredible Growth" and came away disappointed.
I did, but I also read the comment: "While this is an interesting statistic there is no additional analysis that it correlates to python's usage growth" -- which is what I responded to.
>There are many possible explanations of a more indirect correlation that could be skewing the numbers in unexpected ways but none are discussed in the article.
Like what though? Aside from some kind of bias on SO I can't think of any alternative explanation that actual growth.
Now, what that growth was based on, that can have many explanations, sure.
I completely agree that there is growth but the question is how much. To me it feels like the 2012 python statistic is unexpectedly small. It's not like python wasn't very popular in 2012.
To put it another way: do you believe that the first graph in the article directly represents programming language usage percentiles, in the field, over five years? I'm guessing not. I certainly don't.
Here is a graph of programming language popularity on github from 2012-2014:
While this metric is arguably just as fuzzy as any other, the lines in the this graph definitely seem to agree with my gut feeling for language percentiles during that period. I'm not trying to say that my "gut feeling" has any significance. I'm just re-emphasizing that the article didn't give me any useful insight into the significance or meaning of the Stack Overflow statistics. I couldn't find an equivalent chart for the years since 2014 but would love to see one. Either way, it's not like python was ever far behind. It was basically in third place in both the beginning and end of that time period. But in the beginning of the SO chart it's in sixth place. And the SO chart doesn't even include ruby, which is ahead of python in 2012 in the loggly chart. I guess people really don't go to SO for ruby questions? It's certainly possible that the quality of answers on SO for different languages has a lot of variance, which is one possible explanation for the difference. Maybe python questions in 2012 on SO weren't that good. In my experience I'm far more likely to check the python docs before I go looking on SO. Maybe there were changes in google rankings for certain things. Maybe there were certain types of articles that took up the lion's share of growth, like "how to port to python 3", which could have brought existing python devs back into heavy view rotation. Python has also gotten a lot more complicated with things like asyncio, etc., which could again draw more article views but not be directly related to usage growth.
> Anecdotally it doesn't seem to me that python usage in "industry" has grown that dramatically in the last five years, but there could be other factors involved such as education. Or my biased gut instinct could be completely off.
I actually was looking at these numbers just a few days ago, trying to decide on what language to embed in something I'm working on (CPython, Duktape Javascript or Lua), and came to the same confused realization - the number of Python questions on Stack Overflow had increased significantly over the last few years, but I couldn't quite correlate why. I put in a bunch of different related tags, but none of them could account for such a large jump.
trying to decide on what language to embed in something I'm working on (CPython, Duktape Javascript or Lua)
I'm trying to make a similar decision at the moment - have you chosen which of these languages to use? I would be interested to hear your thoughts on the pros and cons of each.
To the article's credit it actually differentiates between first world ("high income") countries and the rest of the world. The growth has been even more in high income countries so that explanation doesn't work for the data in the article.
Yes, I'm aware of that. I wasn't talking specifically about the data in the article but about Python's growth in general. I don't believe the popularity of a tag in a forum is a reliable indicator of the growth of a language, it may simply be this forum becoming more popular among Python users.
Python's rising popularity among the scientific computing community is simply not enough to support the idea of a dramatic increase in Python adoption. As a long-time user of the language I share your appreciation Python usage in the industry has not dramatically increased in the past few years.
On the other hand, there's plenty of evidence suggesting Python is becoming increasingly popular outside high income countries.
The chart there really emphasizes to what degree Java, C++, and to a lesser degree Javascript are used as teaching languages. There's a clear cyclical pattern in their question frequency that presumably corresponds to academic semesters.
Is "the next Facebook" really what today's students dream of building? It just seems a bit... dull. When I was a student we all wanted to create "the next <popular AAA game>".
This was the case in college. We had C++ classes but people learned Python and JS during breaks for internships because the are very little C++ jobs for college students.
I'm a web developer and I love python because it provides me with the fastest and shortest way from an idea to its implementation. In most cases prototype works forever in production.
Rich standard library and clear version support lifecycle are also very important.
I developed with many other languages including java, с++ and Haskell and python is my choice among all.
I agree (Django is life), but I find myself wanting the new optional typing and type checker to mature so I can add them to my set of linters. I'm tired of 1) not knowing whether I made a mistake and passed something of the wrong type and 2) not being able to see at a glance what types each function is supposed to accept and return.
I'm not sure, I haven't managed to write an app that large yet. I'm also not sure how you can outgrow Django, since you can always split off parts of your codebase, or not use the parts of Django you don't need.
I've recently written a flask application too, and I found myself having to reinvent the wheel at every step, and I cursed a lot. Django's ecosystem, both in libraries and in sheer community documentation, is hard to replicate.
Grad student at a math department here- I see python used in tons or research applications with fellow students studying math, engineering, stats, etc. But in my experience Matlab still reigns king in math academia, especially in classroom environments.
I'm surprised by the decline of C#. I thought it was better off than java these days. Also PHP is gliding down. They improved the language a lot, but I can't say I will miss the PHP world much.
The graphs are for # of views to questions on SO involving that language. What it likely means for C# (and PHP/Java, etc) is that the user-base is maturing and stabilizing, and not as many questions need to be asked (or found). For C#, it could also have something to do with the fact that the MSDN .NET docs have vastly improved over the past 3-5 years.
I do my hobby development on Linux, Python 2.7 and 3.x are always installed but 2.7 is the default. What is the future for Linux distros dropping 2.x and moving only to 3.x?
My understanding is that Linux requires 2.7 as the default version, but it means development is a minor PITA, I have to set up v3 and use pip3 (iirc) instead of just pip and type python3 in the shell instead of just python. It appears minor, but it can get a bit annoying for a Python newbie.
Linux doesn't require 2.7, and will be switching soon. But you can remove most of the pain from a future migration by just adding the following to all your 2.7 code and changing your code.
Note that these future modules were all available in 2.6 which was released in 2008. But just like the ruby 2.0 upgrade people procrastinate. But you only have 2 years before python 2.7 is no longer maintained.
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
Searching Indeed.co.uk by title I've noticed Django gradually gain a 10% lead over Rails in the last 12 months. Python:Ruby is currently 2.9 for London and 3.2 nationwide. I think Django's rise is due to the increased adoption of Python generally and also the meteoric rise of Node.js in the same space as Rails.
The French grande école d'ingénieur in CS & applied Maths in which I study made the switch from Ada to Python as the first language teached last year.
Before that I saw all the French Higher School Preparatory Classes starting to teach C.S. with Python3 in 2013.
So I am happy to see that many of France's higher education establishments made the right move.
My hypothesis is that all this caused by data science, machine learning, and deep learning. R is quite cryptic and hard to learn. Almost all new libraries for DS are written in Python.
Also, in favor of this hypothesis is that rich countries are ahead of the curve in DS adoption. It's much harder to find a DS job in a developing or not so wealthy developed country than in US.
I know I shouldn't feel that way, but as a developer who loathes Python, this is a painful read. I start to feel that if I can't fight the trend then I should join it. But then every time I try Python, I feel such unease because of the low performance and the dynamic typing.
It's just something I don't understand... this is really depressing to me.
I am not an evangelist of any form, but running python code with pypy isn't that bad in terms of performance (especially given the ease of writing the actual code), and you can use type hints if you like since 3.5 : https://docs.python.org/3/library/typing.html
I hate Python also - supposed to be this magic low barrier to entry language, but as soon as I want to do anything of substance I start encountering just as much complexity as any other language, only the documentation is worse, and the libraries I want to use are in one way or another a nightmare. I don't buy the hype, use it reluctantly when I have to.
Nim's designer is too enamored with Delphi and has chosen to borrow many of Delphi's bad ideas for nostalgic reasons. This adds more ambiguities, more Perl-like symbol use, more reliance on an intelligent IDE, etc. Python is about the importance of simplicity and readability. I don't see Python devs abandoning Python for Nim any time soon. The ones who would have been tempted by Nim have already been tempted away by Go.
Presumably you are referring to the way imports are done in Nim, i.e. the fact that all symbols are imported into the current module by default (the equivalent of `from module import *` in Python). Nim could certainly follow Python's lead, but that would limit the language: Nim supports UFCS and operator overloading which requires modules to be imported this way.
You can of course feel free to use `from module import nil` in Nim to enforce module name prefixes for each procedure call. But keep in mind that there is no risk of ambiguities, the compiler statically checks everything and refuses to compile your code if there is an ambiguity (at which point you will need to resolve it by prefixing with the module name).
I would be interested to hear more about other things that you think are a bad idea. Could you elaborate a bit?
Do not worry ,still breathing here , when I started Fortran,RPGII and Cobol was the thing and Python did not
exist. Now Python is becoming the old elephant and new
languages like Nim will evolve and they will grow an
ecosystem over time.
It's undeniable that some new languages will supplant the current bunch (except possibly for C; this thing might well be unkillable by this point). But I don't see any particular reason why it would be Nim.
The "killer app" then was an app server + document store called Zope.
Things were smaller than today, but Python was already gaining popularity for scripting. Circa 2001 or so it also had good Gtk and Qt bindings -- there were even some KDE-inspired satellite commercial companies catering to Python/QT.
The discussions in Python land then were about the mythical Python 3000 -- the next, re-imagined version of Python that would fix issues, etc. That was eventually Python 3.
Perl was still king for scripting/admin stuff but losing momentum, and Ruby was mentioned as a nice alternative to Python, but with much smaller use at the time (and even today I think).
> what the ecosystem for Python looked like in ~1999
Very small. The language was not popular and many people claimed a community project could not overtake languages backed by companies.
Same for the Linux kernel.
Thank you for supporting my point. However, based on your nickname I wouldn't be surprised if you were a little biased :)
I would actually love to read some forum/mailing list posts about this. If anyone has any links handy (or knows where to find them) then please post them.
I think about that once in a while, since I've gone all-in on Python, and here's how I've come to grips with it. Even if Python eventually declines (as did Pascal, HyperCard, and Visual Basic), it will still have made me a better programmer, and the continual improvement of languages means that the young contenders have to prove themselves better than Python.
All the scientists I know on Python (N > 30) use matplotlib and (numpy or scipy or pandas). Some niche libraries for certain data formats are also necessary. These libraries are compatible in 2 and 3, and are easy to install with pip or conda, so everyone I've shown them to has made the jump quickly from Matlab, IDL, and others.
one funny way to interprete the data would be: "the language gets more and more complicated and batteries which used to be included now have to be Inquired on Stack overflow."
You have to give props to Python Foundation and just python's community in general - very well managed on both sides.
PEP proposals and the general pythons philosophy is still going strong while new major changes are being introduced in every version without breaking stuff.
As someone who has been contributing to python q&a websites and such for quite a while now, I feel like I'm starting to have trouble of keeping up - a lot of new newcommers but even more people willing to help!
I'd be really be surprised to see Python on a decline any time soon.
This post is suggesting the opposite, that Python is more healthy than ever and growing incredibly fast.