With GPT4 helping with the refactor, there's no reason to start migrating code away from Pandas imo. A lot of people say they think Pandas is fast enough for their needs, but you're literally getting a 95% speed improvement for free.
This is a huge difference in productivity, especially when running code and doing a lot of slicing in notebooks.
Polars is an immense project, and I hope it continues to gain traction. But there's lots more factors than just speed.
The main one in my team is ubiquity- i.e. lots of people know pandas, who might not be traditional "developers". I.e. data scientists, data analysts etc. Having a data scientist put together some code, it gets optimized by an engineer, and they can talk back and forth about the same code is a massive benefit.
Shifting to polars (and keeping that ability to collaborate) would require not just training the engineers to use a new framework, but all the analysts, data scientists etc that they are adjescant to. That's a huge business cost, and in a lot of cases it might be worth it. But I wouldn't describe it as "getting 95% speed increase for free".
While that's fair, it's fairly easy to fit it in only the most intensive operations and then seamlessly convert back to a pandas data frame.
I understand why you wouldn’t do this on an organizational level for production workflows, but for personal workflows in my opinion, it’s a no-brainer to incrementally learn and adopt it.
Where ive seen resistance to polars in Python land its been either "Pandas is already used/standard and people understand/know it" or "If you really need speed, you can probably do it in numpy which should be faster again".
It's easier than ever to do a drop-in replacement for data workflows. The whole decision making process between migrating libraries like this is how much time investment is it going to take and how much is it going to pay off?
This is a huge difference in productivity, especially when running code and doing a lot of slicing in notebooks.