It pretty bad. Just ask it to quote something like the first paragraph of moby dick (i.e. "call me ishmael...") it barely gets past 5 words before it completely gets it wrong. Even the worst normal 4B models get 3-5 sentences.
That’s because diffusion models struggle with memorizing sequences while it’s essentially what autoregressive models excel at.
I’ve tried to fine tune a diffusion model to memorize an MD5 hash and after days of hacking on it, it failed to completely memorize it. Autoregressive models of any size learn it over <1000 iterations.
the headline chart was lazy on our part, thanks for the flag and I’ll update it.
each individual benchmark that is combined into agentic coding index was compared on xhigh between ua / codex / pi, and headline improvement was calculated on xhigh, but then agentic coding index pareto chart by default includex codex max, hense the confusion.
(to finish your thought) Which is important for consumer hardware to be better suited to running these models. Cloud providers are already able to batch as many requests as they want together to improve resource utilization, so they will not see a big benefit from diffusion models.