Have been using this in my terminal, editors, and books for a few years.

Lovely font.

As an educator and author, I feel Molly's pain (Anthropic owes me $60k for pirated books). My course and book sales have dropped significantly. I feel like much of the value in my book-based knowledge is harder to discover in AI agents. (One example might be inspecting SHAP values from XGBoost models and using that as feedback to feature engineering for a linear model (like Logistic Regression)). So I'm afraid more advanced techniques might go by the wayside in the name of delivery speed.

I use AI both in the cloud (I tend to avoid Anthropic...) and with local models.

I've probably written more code in the past year than in my whole career. I can now create what I desire (I just created a telemark skiing game over the weekend based on my Strava segments). I think this has helped my teaching as well. In a recent ML course, I created many interactive examples that in the past were just doodles on my whiteboard, but are now embedded in the notebooks I give students.

I've also taught a few AI courses to clients as well. However, with the inevitable rust/ASM-ification of everything, I'm not sure how long those skills will be valid. I do, however, think that having a human in the loop for ML and data analysis is still important.

The people reaping AI's benefits today generally developed their skills as juniors through old-fashioned struggling in the pre-ChatGPT era -- actually reading docs and articles, trial and error, puzzling over mysterious bugs, and generally doing lots of mental lifting.

How will skill building happen when LLMs and coding harnesses seemingly provide all the answers at one's fingertips?

That's what I wonder. I know to ask the right questions because I've been through the grinder. AI is beneficial to me. But where will the new knowledge come from if we stop learning? Stuck in this year of your deity?

> How will skill building happen when LLMs and coding harnesses seemingly provide all the answers at one's fingertips?

They don't provide all the answers. And are often wrong/mistaken. But they also produce stuff faster than you can consume it. This is great for when you know exactly what you want and can cheaply test the output and guide the LLM to the right solution. It is expensive (in developer hours) in other cases.

I have been building a new PL+VM this year and had to pause development on a few occasions because I have not had the time to go through the implementation because there are so many modules. But smaller tools (5-10KLOC), you can judge fairly quickly.

So, the answer to your question is: people need to discover the strengths and weaknesses of the tool on their own, decide what kind of expertise they want to attain, and if it is worth the effort.

That's a great question. I don't know what the future will be like, but looking into my crystal ball, I'm not seeing much hand-generated code. Very little (or no) human code review.

Having the taste and business sense to develop the right thing will be important. If it stops working, ask the AI to fix it.

> That's a great question.

People have been asking this for a couple of years already. I have yet to hear a real answer beyond "We'll cross that bridge when we get to it."

While this may be true, there are certain types of development (rockets and kernels come to mind) where you will always need highly skilled people willing to be responsible. LLMs excel at many logical tasks but they struggle with creativity and responsibility.

At least, so far.

How will people learn to ride horses, when everyone drives a car? They generally won't, because that is now obsolete.

How does it work when the skill is _thinking_, which is not so much a skill as a basic component of being a functional human?

That's not how the US military operates. The Marine Corps, for example, has an explicit concept of the "strategic corporal".

Perhaps so for the most junior grunts, but officers with people under their command are definitely expected to think.

If you are in the military one of the most important things you need to be able to do is perform what someone higher ranked tells you to. Do not think about it, just do it. Doesn't matter whether you are at the bottom or a major. You are not there to question a command, you are there to execute it.

You can think about how to best execute the command and that's it. Without that you don't have a military.

The ubiquitous "LLM slop is the car and thoughtful development is the horse" analogy makes me think of Steve Jobs's "Bicycle for the Mind".

https://www.themarginalian.org/2011/12/21/steve-jobs-bicycle...

IMHO LLMs are more like "falling asleep on the bus and ending up in the wrong city".

Out of interest, how do you know that Anthropic has ingested your books? Is there a way to prove it? And how do you calculate the compensation figure? Finally, why just Anthropic? Did OpenAI not do the same?

There was a class action lawsuit that listed pirated books

I teach many data/ML courses for clients using their data. It is amazing how many insights both the students (as SMEs of the data) and the teacher (as an SME of the techniques) can have when going through certain techniques together. Quite often, we get a "those last 5 minutes more than paid for the training" insight.

I think that is possible with AI (I haven't experienced it), but I also feel like most folks use basic/naive/cursory techniques when using AI because they don't know what they don't know.

I've been in multiple workshops for creators that were ran by LinkedIn. They have all mentioned not to use links.

BTW Twitter penalizes links as well.

Which sucks when you are for instance trying to let the professional circle know about an upcoming meetup that has a registration link.

Did you ever report these issues to polars developers?

I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)

Yeah, was in the Discord for a while

Most of the time it works, but there are still corner cases throughout the python ML ecosystem where polars fails.

This and the decision not to natively plot with matplotlib keep me in pandas for most tasks. (Plus there's is still a relatively large demand for pandas training.)

Rewriting pandas to polars is relatively trivial for most tasks these days. Especially if you wrote your pandas code correctly.

I still prefer (and use) Pandas for EDA. I think matplotlib integration is a better choice for most viz.

Also, I'm probably in the top 3-5 worldwide for number of folks I've trained with pandas. I offer Polars training and there is little demand for it.

Use pandas if you want advanced analytics, visualization, or ml.

Use SQL if you need to move data around.

How would you use sql to move data from three excel files (some of them may have arbitrary number of worksheets), pack of json files, two csv files, one mysql, one postgres db, some parquets, some of them in S3) ???

And in the end save processed data in aws s3 in another format like iceberg or whatever

In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)

Those are generally rules of thumbs for data pipelines. (Plus duckdb can read Excel these days I think).

If you learn to write pandas correctly you end up writing it very similar to polars (or tidyverse).

I agree that the API has warts, though typically it is more concise than polars.

The problem is a small change in the question can force you to make huge changes in the code in pandas. I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas.

[1]https://bjarkehautop.github.io/Website/blog/data-wrangling-t...

Personally, I think the inclusion of lambda functions in the pandas code shows that the approach isnt ideal to begin with. Fine on smaller datasets, but typically becomes a slow and cumbersome overhead when interacting with larger datasets. More verbose per line vector operations are generally orders of magnitude faster, and become more so as the data grows. They also tend to be easier to read / identify what is going on or what the intent is for people unfamiliar with the code base.

This is a common complaint I get all the time (heard it this week while teaching pandas). I compare this to whitespace indentation in Python.

Lots of folks complain about it before using it. After they use it it is a non issue.

If it really is an issue (and it generally isn't a cause of vectorization removal when used correctly) and you can't get over the syntactic noise if the lambda, pandas 3 introduced pd.col (that work in most (I filed a big about some exceptions) places when you'd use lambda).

Ive seen it in such a wide variety of scenarios where it causes vectorisation removal because of its implementation that I dont reach for it by default any more. I've a number of devs around me who have similar opinions. Each know how to use it, but you dont always write the code you're reviewing or optimising.

Would love to see examples.

I looked at your code very quickly, but it looks like you need to use .filter after a .groupby...

The text right above the code says why you can't...

edit:

Let me clarify. From the blog-post:

> since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it:

Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.groupby().filter(lambda: )`, since you need a row-wise decision.

I misspoke, you need to use .groupby/.transform to add a new filtering column:

    (sales
      .assign(country_median=lambda df_: (
          df_.groupby("country")["amount"].transform("median")
      ))
      .query("amount <= country_median * 10")
      .assign(net=pd.col('amount') - pd.col('discount'))
      .groupby("country", as_index=False)
      .agg(total=("net", "sum"))
    )

Yes, so basically equivalent to the code I showed in the blog.

I'm confused, you can use filter after a groupby in pandas...

It's late here, I'm going to bed, perhaps I'll write the code tomorrow when I'm at my laptop and not on my phone.

Most folks just need to learn how to use pandas well and that will open enough doors. Then they can move to polars or duck if needed.