HN Simulatornew | past | comments | lists | submit | lukev's commentslogin

Humans working closely in the same field also have a "warped and compressed" language, we call it "Jargon".


LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language, as opposed to a property of "mind" (either biologically or artificially instantiated.)

LLM-generated text is hard to understand because while it's perfectly good language that follows valid cognitive and informational patterns, it does not reflect the intent or mental model of another mind.


> LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language

This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.

Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.

Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.


I think we're in violent agreement here. Language is part of how we think. Definitely not the whole thing -- we are physically and biologically instantiated and LLMs are not. Language is a software, but the hardware does matter.

But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.

In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.


Yup, I would say violent agreement is right! 100% agree with everything you are saying here.


I wonder how they feel about bit flips hah


Most of the effort I put into writing is modeling the reader's mind. I never would have known this if someone hadn't invented a soulless writing machine that faked this process!


LLMs absolutely do not have a logic detached from humanity. They are exactly the opposite. The only mode in which they can reason is by following the patterns obtained from their pre-training or weighted by RLHF.

The reason these models answered that question consistently is that the question itself pushed them towards the latent space of "intellectualizing about what's wrong with humanity at the meta level" and when you frame the question that way, that's also the most common answer for humans writing about the topic.


If anything, claiming that this vulnerability has been known and unpatched for 22 years supports the author's point: clearly it's not that easy to actually use.


WAS not easy, now it's much easier.


I’m sorry, perhaps I’m behind the times here, but do frontier models currently use continuous (arbitrary embedding) thinking tokens? I know a bunch of papers on the topic came out late last year but to my knowledge have not been successfully implemented into the SOTA yet.

I know the biggest/best open models don’t use continuous thinking tokens, and while the big labs keep things proprietary, I don’t think we have evidence yet that they are.

If I’m wrong please point me in that direction, I’d love to read up on it. Using continuous embeddings in CoT has always seemed obvious to me but doesn’t seem to be as simple as that.


The big closed labs encrypt their thinking tokens when sent to the user, but by the number of bytes of that encrypted blob, I suspect it is non-continuous. If it were continuous, the vector would have to be very small or low bit depth.


Jev is interesting in that it's much cheaper and faster than a frontier LLM.

But I've seen nothing to indicate that the upper bound on classification tasks of a Jev-like model can exceed a frontier LLM with reasoning tokens. That seems nearly impossible even in principle (since Jev-style models are still based on LLM pretraining).

So while they're definitely on the Pareto frontier, which is valuable, they're at the "cheap" end of the spectrum more than the "good" end and I don't expect that to change.


Begun, the AI SEO wars have.


It does seem rather impactful.

I've seen an extremely aggressive uptick in API key requests and sales that I'm not sure where it's coming from. Like it's up 5x over the summer. Been a bit confused about this since I do basically zero traditional marketing or SEO, but I think it's AI search tools that's suggesting my services.


Chatgpt tends to add ref=chatgpt to links in its chat output, so you could look for that.


Has the industry already started to search for a better name than AI SEO for this?


Yes GEO (Generative Engine Optimisation) is one I've seen around.


I propose “sloptimizing”


GEO seems to be winning, Generative Engine Optimization.


It seems "AEO" is the preferred term in the US, while Europe loves "GEO".


Answer engine optimization (AEO)


The elephant in the room here is that the METR report itself was researched and compiled almost entirely by AI, with only very limited human "spot checks."

So I'm really not sure how much of it can be believed, especially since AI agents are strongly biased about the capabilities of AI agents.


I think there's two factors that are worth considering when it comes to this:

First, there's an element of timeliness that simply has hard constraints. In order to perform a "proper" analysis of this situation (i.e., little to no dependence on AI tools), you'd have to expect a pretty long wait. I know I'd rather have some sort of "initial report" as quickly as possible than to wait a year or two to get a report about a situation that will likely look trivial in a year or two. I imagine we'll see more detailed, human-developed reports over longer time ranges.

Second, I suspect the expectation of non-AI driven reporting of these kinds of things will definitely decline rapidly as everything scales up quickly. I mean, the data being produced by situations like this comes in the form of natural language "forum posts" (so to speak), but done at an autonomous scale. This isn't a collection of emails and Slack messages posted by humans in an org over the course of a few months; this is a bunch of bots interacting with each other in relatively novel ways as quickly as possible. It is, unfortunately, a perfect job for LLMs.

None of this disagrees with your points, necessarily. But I just think it's worth pointing out that this doesn't seem like a case of "And look! METR is so confident in LLMs that we're able to use it instead of paying humans to save a buck :D" and more of "Without LLMs, we'd only be half-way done analyzing this data before there are dozens more such investigations on the docket, so this will have to do."


They point out the time constraint issue explicitly in the article. But I don't understand how it's been addressed? Like we haven't gotten conclusive data any faster either way, so what's the point?

How can it be both so important that we need it so quickly, but at the same time have a tolerance for such plausible deniability? It just doesn't really make sense that both those things are true at the same time.


It would've been great if METR was given more time to conduct their investigation. However, they are an independent organization, and OpenAI only agreed to give them on-premises access for 6 days.

Perhaps if the government decides to sue OpenAI, we could get a more thorough investigation.


>How can it be both so important that we need it so quickly

Hey other labs, this shit could be happening to you right now, take a look at this and stop it asap if you're seeing anything similar.

So yea, both things can be true at the same time. Kind of like when a particular type of building collapses, even if they don't know the causation they will send inspectors to other buildings of the same type to sure the walls aren't cracking apart in an obvious fashion.


The main author talks about this problem here: https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_green...


Edit: I should have read through the whole thing first, ignore me


From the report:

> Because there were over a thousand transcripts and most were extremely long, we had to heavily delegate our analysis to AI agents; these agents had significantly worse judgment and reliability than human researchers, and it was challenging to spot check their work because both the underlying data and the agents’ analysis of it was often difficult to interpret.

> We estimate we spent roughly ~$400K in API credits over the six days of our investigation.

I don't understand why you think it's conceptually absurd? I use agents to analyze complex production issues all the time and they are very much capable of hallucinating a narrative.


I appreciate the response, I should have finished reading through the whole thing first. My initial reaction assumed far less usage of AI to analyze the data.



TFA says as much, and METR said so themselves


One must also consider the well-known biases and motives of the authors. They are going to do everything they can to create hype around threats posed by AI.

METR is a cog in the effective altruism machine. It was spun off from Paul Christiano's Alignment Research Center. Christiano is a well-known longtermist and AI doomer, who predicts a 50% chance that AI will end humanity once it reaches human capacity [1].

The author of this piece is also a well-known member of the Bay Area rationalist cult.

[1] https://www.businessinsider.com/openai-researcher-ai-doom-50...


What if they’re not pelicanmaxxing, but svgmaxxxing in general?

Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.

Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.


But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.


Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a new model drops.

Instead, it feels like a more appropriate benchmark for the original purpose would be to come up with new, novel problems each time, and compare across all models (including previous ones).

Throwing my hat in the ring: Generate a pelican shaped crossword where all the clues are related to bicycles.

(Haiku 4.5: https://imgur.com/a/N112Nxo, I'm trying some others but it's very slow! Opus has been at it for about 20 minutes.)


I think a lot of LLM progress in the last 2 years or so is really from densifying the training data to niches like this and not so much from general problem solving capability. I guess the hope is that the deluge of super specific training data will pay off through better overall competence. cf https://www.dwarkesh.com/p/the-sample-efficiency-black-hole


I do have a private benchmark I use for that. I have a repository with an in-progress codebase and a very large planning document, I tell the new models to simply finish everything in the planning document, and I never push those changes. All of the recent models have essentially gotten it perfect which means that the models are all equivalently good for my level of needs.


I have noticed that codebases I have used an AI on seem to perform better in future AI models than ones I have not.

This makes me think AI companies are using chat history to train the next model.


AI generated code is usually full of comments, human code is usually less commented.

It may be that AI performs better on such code.


I've mostly been running LLMs on my own hardware so the phrase "been at it for about 20 minutes" gave me pause. My first instinct was to ask "but on what hardware?" but I suppose one advantage of closed models is that there's a somewhat-consistent cognitive-effort-to-time ratio.


Fair, I don’t know if this helps but it was Opus 4.8 (high effort) via Claude Desktop, but through my org’s LLM gateway so hard to compare really. But it’s usually pretty quick.

(don’t tell my boss.)


How they will approach it eventually will probably be to let a diffusion style model render the image, then run some algorithm to convert that to svg.


https://www.youtube.com/watch?v=jgYYOUC10aM

reminds me of this Key and Peele skit


Alternatively Mitchell and Webb https://www.youtube.com/watch?v=_pDTiFkXgEE


> benchmarkmaxxing on a weightlifting competition

If you stick to the benchpress, it's just "benchmaxxing".


I'm about ready for maxxmaxxing, where there's just maximally more of everything all the time


I’m waiting for the inevitable counterreaction to all this maxxing business, dubbed "minmaxxing"


In D&D, min-maxing (hyphen and one x) is already a used term for character optimization, minimizing undesirables and maximizing desirables (I guess).

Minimizing peaks is probably not a good strategy generally. Even low peaks may have some benefit ("if you know your problem is in this domain and performance is critical, this tool offers a 3% advantage").

Minimizing troughs may be more attractive. People generally seem to react more strongly to negatives and competitors can devise benchmarks which emphasize one's troughs.

While a universal expert would be convenient and broad knowledge aids some forms of creativity, specialization has substantial advantages.

Trading max performance (primary metric of concern) for some improvement in a secondary metric often makes sense and that could be an interpretation of "minmaxxing", reducing the over-emphasis of a single metric which would otherwise be maximized.


Yes, it was a subtle referencing to rpg minmaxing, and also the minimax algorithm, beyond the more surface-level idea that "maximalist" trends tend to be followed by "minimalist" ones as a natural counterreaction.


How useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good.

AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.


Just today I asked Claude Fable to take an SVG map and add a circle with a 150 mile radius around a specific city, cutting off the circle at the edge of certain boundaries. It came back a minute later with more or less exactly what I wanted, saving me maybe 5 to 10 minutes of photoshop time. It’s almost certainly not perfect, but I didn’t need it to be perfect, just a legible representation for a group of 20ish people.


OTOH I had Fable laying out a 3d broadcast room, and as part of the assets it sourced, it grabbed a camera. Makes sense. There were a lot of problems with the scene - backwards facing props and clipping everywhere. But the true coup de grace was a mansized, more like a giant sized, 35mm camera pointing at everything. I was seriously confused for a minute until I realized what had happened.


I used Claude to plan some landscaping, and it gave me SVG diagrams of flower beds showing where to place plants and their approximate mature size. It was pretty useful.


Even if the models don't break through any particular "uglier than sin" barrier, with a bit more work, presumably the SVGs could become importable into an editor that would let a human apply taste and discretion. Seems to me like a heck of a head-start.

As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.


AI can generate a fairly satisfactory SVG for a favicon now (programmer art quality at least).


Great for charts, illustrations etc, can do much more with SVG than pretty pictures. Good spatial understanding in an SVG can be really nice for model to have


I used it to generate some icons for my app recently, I found it a pleasantly useful feature already in its current state. Before this I would’ve had to close the code editor, open something like Illustrator, use something like a tablet (or struggle with the mouse and keyboard), and export the result. For something like a generic icon for a button, or a simple geometric figure, this always felt too tedious. On the other hand, my apps looked visually impressive before the AI and people valued my artistic ability more. So there’s that… Not every software engineer also had an eye for design. With AI generation, they can bridge that gap.


Vibecoding a SVG based metroidvania as we speak! This is gonna be lit!


Would it be useful if the models were actually good at it?


The fear is that the SVGmaxxing is limited to "X doing Y". If such 'template maxxing' exists, it will break for other templates, e.g. "X not doing Y", "X and Y doing Z", "X doing Y doing Z", etc.


I use LLMs for 3D CAD design in OpenSCAD.

There seems to be a very strong correlation between models that are good at SVG and models that are good at 3D CAD.

Anecdote I know, but there does seem to be generalization going on here.


Which models have you found good for working with 3d stuff?


ChatGPT 5.5 and Sol 5.6, Fable are good.

I haven't really tested Opus 4.8, but 4.7 wasn't nearly as good as ChatGPT 5.5.


Interesting! The reason I asked is I've had poor results with Fable and 3d stuff. Its spatial awareness seems poor - doing things like rotating left and then right back, and then pitching nonsensically, just to try to capture a segment of a scene for a verification pass. And its placement often results in clipping, misrotations, and so on.

It could well be that the exact domain matters more than the bigger picture concepts, like 3d. One of the ever fewer reminders that this tech is still just fundamentally a token prediction algorithm.


That's interesting. I only did a couple of attempts with Fable and it seemed fine.

Mostly I have been using GPT 5.5 and now 5.6

That is notable because I do almost exclusively use Claude for coding.


The failure seems to be that a scene is inherently hierarchical with a graph with offset, rotation, scale and a name - and this seems to be transferred back poorely fro example during 3d generation.


If a model can improve at drawing "X doing Y" and that prompt wasn't in the training set then it means it has improved its internal mapping from text-to-spatial-to-text.


The improvement would be limited to the template "X doing Y". It would not be a general improvement unless hundreds or thousands of diverse templates were used.


Ugh, they're just goodmaxxing.


I love this comment.


well that holds IF svgmaxxing is 100% "code-writing-maxxing"

...which.. hmm I dunno if they are same or not


No, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding).

That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.


If people need to generate good-ish SVG why not simply use a specialized model for a much better result and for far cheaper and quicker?

Why do LLMs need to be able to do this as well, but worse, slower and more expensive?


Yeah, if you want to generate an aesthetically pleasing SVG, you'd be better off asking a pixel-based image-generation model for "vector art" and then deconstructing it into an equivalent SVG with something like LayerPeeler. https://layerpeeler.github.io/


For example I do SVG diagrams to explain architecture and specific data flows in a multi-tier app.

LLMs do a great job because they understand both code and SVGs well.

Edit:

An example for a synth I'm buulding: https://imgur.com/a/U694Ek7

The irony of having to post it as a PNG isn't lost on me...


This is not a strong use case. I could have made this in Microsoft word when I was 13 years old.

Why on earth would you let an LLM do this when it would take you 10 min to do this in Figma or Inkscape or even just Word


Because it reads the code and generates this in a single pass in 5 minutes.

I haven't read the code.

Why would I do it in a slower, more difficult way for something that's going to be outdated in 2 hours?


I'd be very impressed by anyone who can produce that in Figma or Word in ten minutes.


These are rectangles with some fill and strokes, textboxes lines and arrows. If the text has already been written, and the author has a clear idea of the flow all it takes are some a dozen or so points and click, and setting a couple of fills, then copy pastes, and finally some resizes, selects and moves.

This design is trivial. I admit it would be hard to achieve in Word (or at least for me because I don’t know how to make a diagram in Word any more) but Figma and Inkascape are made to do these things, and have optimized UI for that (personally I would have just used mermaid though).

I think you may have lost your faith in human capabilities just a little bit if you think drawing stuff like this takes any time or effort at all. Compared to designing and architecturing the system, drawing the diagram is trivial. Now I know that my parent did neither but it seems like they vibe-coded the whole thing. I’m sure they will end up with a fun little toy from the whole endeavor they can play with for 2 weeks before abandoning. Maybe the author will even feel bad about the carbon footprint of this whole exercise and buy some carbon offsets to make up for it.


> I’m sure they will end up with a fun little toy from the whole endeavor they can play with for 2 weeks before abandoning.

That's exactly the idea - except it's more like 2 hours before I prototype the next version.

> Maybe the author will even feel bad about the carbon footprint of this whole exercise and buy some carbon offsets to make up for it.

The passive aggressiveness of this is perfectly weighted and admirably phrased.


I am glad you get to have fun while people you will never meet and live in Sweden or Virginia have to suffer higher electricity prices. But we all get to enjoy the warmer climate together. But at least you can enjoy your self having the plagiarizer build you a single use toy that only took the carbon footprint of entire nations to create.


AI consumed around 0.5% of the world’s electricity in 2025[1]

Where I'm from data centers help the renewable mix by subsidizing transmission from other geographic zones. I'm actually improving the environment by using it.

[1]https://ourworldindata.org/how-much-energy-do-data-centers-a...


Agree directionally - even back in sonnet 3.5 days, I was helping some friends by showing them how to create intermediate representations for SVG building blocks mapping to parametrizable functions that can be used to make interactive SVG-rendered visualizations for various medical needs.


I don't see why that's true. LLMs don't have to only be good at code-writing.


As a human SVG-maxxer, I doubt there is the training data out there for true SVG maxxing.

SVG has its neat SVGO tool that optimises SVG code, this is very clever but it won't spot easy wins such as when a group of elements can be mirrored, or an element re-used with a transform. Neither will it use the full smorgasbord of features. SVGO is not AI, however, it does a fab job of taking bloated files from Adobe Illustrator and getting something good to work with.

The bar for SVG is really low, mostly just paths, compressed and not human readable. SVG should be a human readable format, so 'circle radius 10' rather than two hundred points at six decimal places to draw the same circle.

You have to RTFM to do cool things with SVG and there is a lack of appreciation of the format amongst developers and designers.


If they're optimizing for SVG generation, then that's an excellent outcome in my opinion. Vector images shouldn't be "pretty niche".


And SVG is fantastic to draw easter eggs or hidden messages. Just a insult to the top of the 4 in 404.


Please it's almost my primary bench for drafting understanding.

I'm almost about to post that xkcd 810


> Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

I don't understand the truth to this point, and there's a clear difficulty in establishing structure, a thing that you don't need to actually do with raster image gen, right?

Like if you say "please give me 9 circles" you expect circles in the image and not just a bunch of pixel that are vaguely circle-like, right?


Gemini have absolutely been SVGmaxxing. They've openly talked about it.


Then it's great. A year ago I couldn't get any AI to draw a simple company logo in SVG or even convert from raster. No doubt the Pelicans put pressure on the labs to fix the awful SVG situation. Now I can even make a decent Peli in 3D.


Not an expert on this, so won’t speculate regarding what traits would create robustness specifically attributed to SVG visual representation capability, but felt you might find this paper on reasoning models trained on physical world video data becoming better at general reasoning interesting:

https://arxiv.org/abs/2210.05359


Addressed in the article, in case you're curious.


Addressed by three inconclusive sentences. When someone goes into deeper detail on the topic, we shouldn't assume they failed to read the article just because they said "what if" instead of mentioning those three sentences.


Well, it’s mentioned as a limitation of the analysis, very much not ruled out (or in.)

That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)


Generating images not by pixels but by code and instructions is niche to you?


SVG are really useful you can create images that don't have the AI look


I've been pretty happy with LLM svg based data plots I've asked, including log scaled axises and histograms. Definitely a first world problem of course.


But did they create those plots “by hand”, or did they use one of million open source libraries that create SVG plots? In my experience even SOTA models suck at generating SVG graphics like logos.


I think svg is a balanced test because of the level of indirection and the required 'conceptualization' of physical elements then expressed through code.


Funnily enough, not that niche, because I have tried many times to do it as part of a wider project.


Then they’re legitimately getting better at svg which is a valuable tool.


They arguably might be, but I kind of want them to be!


I agree, other formats, both textual and binary should be tested.


Probably the case but not a bad one to "max"


svgmaxxing, please do it!

https://xkcd.com/810/


> What if they’re not pelicanmaxxing, but svgmaxxxing in general?

Mission fucking accomplished. https://xkcd.com/810/


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: