I'm not following the trends closely, but has Polars become a full replacement for Pandas? Are there use cases where one is better suited than the other?
That's a shame, because Pandas has a really quirky/legacy-burdened API and Polars is super clean by comparison. As someone who had Spark and Pandas experience before switching, Polars felt like Pyspark without the added mental overhead of needing you to think about multi-worker-node parallelism
Polars is effectively a full replacement for Pandas for 99.9% of all cases. The only exception I'm really aware of is if you're working with geospatial data, as there isn't yet a "Geopolars" equivalent of the commonly used "Geopandas". However, Geopolars is still in active development and should eventually be production ready.
Imitation is the sincerest form of flattery! GeoPandas — and its underlying libraries of shapely and GEOS — is an incredible production-ready tool.
GeoPolars is nowhere near the functionality or stability of GeoPandas, but competition is good and, due to its pure-Rust core, GeoPolars will be much easier to use in WebAssembly.
This is awesome!! I'd looked at the project only a month or so and it appeared abandoned, but I must have missed the off-main-branch development going on!
There is a geospatial package for duckdb though which is pretty slick. It is actually how I first learned of duckdb. We were dealing with nationwide parcel datasets and need to apply transformations nationwide and save out to more parquet files. It was easier and cheaper to replace all of the pandas workloads with duckdb.
There are a few packages that still expect pandas inputs and return pandas outputs, so I just convert my Polars dataframes before and after passing them to such a function.
I almost never use pandas directly anymore. Polars is just too good.
Pandas is better for slight in-place or per-row modifications, for loading from less conventional datasets, for transposition/more nuanced row-based aggregation/multi-axis manipulation, for performance when multiprocessing can be used, for interoperability with other libraries (e.g. plotting and statistics)... When you need to operate on huge datasets, use DuckDB, because its performance is even now on-par with Polars regarding speed, while handling huge or more complex joins is a huge win for DuckDB because it better offloads intermediate results to disk, while Polars just dies on me. I haven't tried such joins in Polars 2 though
Yes and no, its not replacing the reason why pandas was popular ie data scientists, but it a full replacement of its pipeline usage, And I would saw also beating out spark
A lot of our came from non cs backgrounds with a lot paper writing, pandas and numpy makes sense to them, its like writing Latex for them, polars is not. Its changing recently as we move towards more AI code but thats how/why things used to be.
my understanding is Polars is faster, scales better without using external solutions, better API, +Rust. Pandas wins if you want to use what the vast majority of folks are using and have used in the past. Probably has a more complete set of helpers / recipes for the little things you bump into when using it thoroughly, but in the age of LLMs, I think that's minor.
>Pandas wins if you want to use what the vast majority of folks are using
Vast majority of skilled developers are now using Polars, unless they are constrained by lack of Narwhals support in their third-party library of choice (e.g. Great Expectations, SHAP). That's the more important trend to follow.
I also found myself reading the manual all the time.
I have a toy project I use for learning different tools. Given a CSV containing every MLB game ever played, calculate the win percentage for each team during its first season.
Pandas makes that kind of simple thing painfully unintuitive. Tools like Polars and dplyr and even SQL require code that's far closer to how people think.
Learn SQL and interface with Duck. You will be 100x faster than Pandas/Polaris duo at fraction of memory. Also SQL is supported literally everywhere with a much more capable than Pandas API. Duck outputs to a Pandas Dataframe, but just treat that like a dictionary. Do all your processing, filtering and aggregation in Duck.
Also, try fx.wtf as a replacement for jq. it comes with a in-built tui viewer that supports vi-keybindings. Ecmascript is built into fx.wtf so you can query the JSON with JS notation (where JSON was born). You can use any JS functions including map/reduce/filter or perform any kind of transformation instead of learning jq dsl that you will forget tomorrow.
There are awkward things. For example, if you ingest a nanosecond resolution timestamp, there's no way to re-export that out of the Polars dataframe with nanosecond resolution.
I'm not sure this is true e.g. you can specify schema `pl.Datetime("ns")`. That survives roundtrip to/from parquet in my experience. True though that the default is `us` whereas pandas defaults to `ns`.
It kind of reminds me of Dungeons and Dragons, in a sense. Pandas is so popular that people learn/use it because of its popularity, rather than because it's the best for any one use-case. Polars is cleaner, faster, and can easily be scaled up for production use-cases so your little POC script for local analysis can be productionized really easily, but there are holdouts still on Pandas because it was so complicated to learn with so many little extra rules to learn to avoid paper cuts that they feel like learning another data manipulation tool would be really hard.
DnD does the same thing: it's the most popular but its rules are this awkward hybrid of legacy cruft and some modern ideas, so learning it a huge effort, which means most people who play it aren't willing to try any other RPG systems even though most of them are dramatically easier to learn because they were built with a clean design from the ground-up.
It's the sunk-cost fallacy as applied to learning something complex, combined with something like the horn effect (inverse of the halo effect) making any competitors look equally complex even if they're not, causing long-time Pandas users/DnD players to strongly resist even looking at other options. Basically the frustration of learning these older systems seemingly traumatizes some people into never straying.
> but there are holdouts still on Pandas because it was so complicated to learn with so many little extra rules to learn to avoid paper cuts that they feel like learning another data manipulation tool would be really hard.
Totally unrelated but I love and dread how that describes almost to the perfection the sad state of affairs we're at with git.
From experience I'll say one thing that Polars doesn't have great interfacing with is doing things like string concatenation, like taking multiple columns and combining them in with static string text in complex ways to create new columns.
Pandas has a really simple ability to just define a new column with
`df['col_a'] + "text" + df[col_b']` where "text" can be any string text inbetween your column values from col_a and col_b
If i remember correctly while you can do pl.col("col_a") + pl.("col_b") for plain concatenation, you can't mix in static text strings like you can with pandas and I haven't found really elegant ways to do that personally. Whereas I've found polars doesn't have as simple of a way to do that. You can choose to add one separator and make that anything you want, but only one separator and only inbetween the two values (so no suffixes or prefixes for example).
That being said, I hate everything to do with the pandas API (especially with its indexing system) and really prefer the more polars API for anyone coming from a SQL or database background. Pandas really shows its sort of academia background rather than a data engineering origin.
I'm not sure what you tried. Is this not what you want?
>>> df = pl.DataFrame({"x": ["a", "b", "c"], "y": ["d", "e", "f"]})
>>> df.with_columns(new=pl.col.x + " text " + pl.col.y)
shape: (3, 3)
┌─────┬─────┬──────────┐
│ x ┆ y ┆ new │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str │
╞═════╪═════╪══════════╡
│ a ┆ d ┆ a text d │
│ b ┆ e ┆ b text e │
│ c ┆ f ┆ c text f │
└─────┴─────┴──────────┘
I'm a little out of touch with these technologies (sveltekit, nextjs), but is there any advantage to using the same technology for both frontend and backend development beyond the rise of full-stack developers?
What about using React/Svelte for the frontend and a separate backend in Node.js or another stack? Is there any real advantage to having a single monolithic codebase for everything?
The main benefit is that it's very easy to have type safety and consistency between your client and your server. NextJS and the like make it feel almost like you're just calling a native JS function in your client, but it's actually doing lots of magic to make that work. The end result is your client is never out of date with your backend API, everything is type safe, you get full type hinting, and of course all of that helps LLMs write better code for you as well.
As long as you can get an accurate OpenAPI schema out of your API, it’s not much of a value add to have the same language throughout. Stuff like Orval and OpenAPI TS make this easy.
You can enforce keeping them in sync at CI time. This is more effective in a monorepo.
OpenAPI does a similar job and is great if you're using different languages, but I disagree it's not much of a value add. You can send way more types between client/server in e.g. NextJS without thinking about it like lists, dates, promises, etc. OpenAPI can't do that, and OpenAPI also needs a codegen step and then you have to work with janky generated clients.
I’ve been working with purportedly janky generated clients across an FE and BFF monorepo and it’s been painless. From where I’m sat, the benefits I’m missing aren’t significant enough to decide the stack.
Between OpenCode and Openrouter which one would you suggest? Sometimes I have pure grunt work to be done on non-sensitive data for which I want to use Chinese models. For example, tasks like extracting something from publicly available large pdf files.
opencode with model inference on cheaperinference.com has been working well for me - glm-5.3-flash is shockingly cheap (i've spent a total of a few dollars over several weeks of heavy usage), fast and capable for cyber tasks
> They are open source developers, giving away free software as a gift. There is no duty here.
But actually there is! They didn't start from scratch. They took another very successfull and beloved project with the promise "to make it better". There is some promise here and the duty not to make it worse.
"Better" is subjective. They made it better...for themselves. Then gave it away so people who share their opinion, had the opportunity to choose to use it.
reply