If found that stack languages are a massive weak spot for LLMs. It requires accurate tracking of what is on the stack and for some reason that doesn't really work. So even the best models end up making the same basic mistakes over and over again.
My theory is that the LLMs can only think in terms of data structures of a certain "dimensionality".
So for example I tried to have LLMs write xml parsers from scratch in Q3 2025, but they kept getting confused and midstream would migrate from c# to xml (that is their solution was half c# half xml file). So thinking in a tree data structure was beyond them.
Then Q1 2026 they were able to write parsers including parsers for languages where I specified with a bunch of examples.
But I tried specifying a language that I thought would require storing the AST in a graph structure. Everytime the LLM would instead parse into a tree and then do post processing on the tree to get the right output. So perhaps graph structures are (were?) beyond them.
I haven't investigated forth for a while, but it's been on my to-do list. I'll have to bump it up some because it sounds like I can kill two birds with one stone here.
I do kind of wonder if tree space is a local maximum, though. You can do quite a lot of things in that space and most of the code that I've read that leans on some sort of graphical thinking (like heavy usage of global variables that effect the execution path all over the call graph) is a less than ideal substrate to have to think through.
[One of the problems with investigating this is that I can't just ask it to solve a problem that is obviously just dijkstra because that's in the training data.]
Interesting. LLMs seem to now handle other kinds of state changes with ease, so I'm not sure why they would trip over this.
Do you mean stack-based source code languages like Forth or PostScript, or directly writing machine instructions for a stack-based ISA/VM like the JVM?
They are also languages which humans basically won't use directly, and have almost zero representation on GitHub etc.
If the training set were loaded with Forth, LLMs would be knocking it out of the park.
In reasoning mode, if Forth were in high demand, I expect a modern LLM would blast out a python script to help it reason through the stack. That's how they seem to handle most problems that they don't innately understand.
> If the training set were loaded with Forth, LLMs would be knocking it out of the park.
That is what I thought initially as well, but I'm no longer sure that that is the case. Forth requires a mental model of what is going on on the stack and that mental model can get pretty complex. There is a lot more 'scope' in Forth than in most modern languages, you can reach pretty much any memory location and fetch any word from the stack, effectively the whole machine is in scope.
It has some interesting properties though. I recently revived a Forth I wrote in 1988 to bring it up to full ANSI compliance and it was a lot of work. It also gave me a renewed respect for the incredibly small footprint that can get you a working forth environment. Originally the program was ~900 lines, and it probably could have been shrunk quite a bit further. But there are a lot of not-so-elegant parts of the standard that caused some bloat.
