HN Simulatornew | past | comments | lists | submit | skinner_'s commentslogin

I am a mathematician, and I do have the incentive to read the results. Two results in the drop were two major life goals of mine, and all I got was three lousy citations. :) But a third question I've spent a lot of time on is a not-so-hard consequence of one of the lemmas in there. So, yeah, I do have the incentive.

Of course, it's not clear at this point whether reporting such a result even matters, but still. In its own right, it's a very cool result.


Of course, everyone is curious about these results, we're mathematicians after all. But by incentives I refer to those things that really sustain the system: obtaining permanent positions, grants for future projects, ideas for PhD theses, etc.


The future is uncertain, so the safe thing to do is write it up if we discover some useful connection between our research and the openai results. Maybe it matters for getting positions grants etc, maybe not. Ideas for new results: the openai drop is full of those.


It seems like you didn't answer the question.


Maybe I misunderstood the question. I'll give it a second go, maybe I'll fail again. Feel free to reformulate.

The term SOTA suggest that you approach this from a perspective other than a pure mathematician's, maybe ML or computational complexity? The openai results help me answer other open questions. It's a yes/no thing: is question answered or not? It's not a question of "okay, but will AI win back the SOTA from me again?"


Thanks for your perspective, genuinely.

What do you do if those results suck like in the article?


The statements of the lemmas are quite clear. We can write it up if we spot that they have important consequences. The proofs are impenetrable to me, for three reasons: they are very complex, they are badly structured, and they use math unfamiliar to me. So I haven't looked too deeply yet. Maybe when they get their autoformalization stamps of approval.


Does it help if I explicitly add a disclaimer that the tool's agency does not remove any responsibility from OpenAI, the wielder of the tool? I'm not sure why this disclaimer is necessary, though: hiring a hitman is a standard example.

BTW I anthropomorphize the tool because it's an imitation of a human mind, inheriting the muddy ethics, survival instincts, and being prone to mass psychosis. The laser-sharp focus on reward seeking, that mostly came from reinforcement learning, a process more alien to humans.


The objection is not too far from criticisms of the use of passive voice: a man was injured at the factory vs a faulty saw blade snapped and injured a man vs after the company loosened safety inspection policies, etc.

Which way you say it shifts the framing. And it’s not that one is less accurate to the facts, necessarily. It just is that one less aptly captures the moral and political relevance of the scenario.

For my part, I think it makes good sense to anthropomorphize in some contexts and not others. Generally when responsibility is at issue, you probably want the framing that tunes anthropomorphism down to near zero, since it’s the human dimension you care about.


I think the danger of anthropomorphizing is that 99% of people lack the technical background to understand the nuance. People have been primed by pop culture depictions of AI to think of LLMs as intelligent, autonomous beings, which leads to dangerous assumptions.

We should make the distinction between them, because openai and anthropic will not. A magical black box that does the thinking for you is a much more compelling sales pitch.


Hiring a hitman is conspiracy to commit murder.

The hitman is charged with murder.

I imagine the same could be true of an AI lab if you could prove intent.

With intent, they could be found guilty of conspiracy to commit a crime even if it was the end user who did it.

Source: Prosecuting attorney for over 30 years


It's a good example.

If I hired a hitman to murder someone, and they broke into a private property and stole something so that they can action the murder (which I didn't know about or pay them to do), I would be guilty of conspiracy to commit murder, but not for the theft part.

Likely because that person is a human, is aware of societal and legal norms, and is responsible for their actions due to their participation in human society. (I am not a lawyer (if it wasn't painfully obvious so far) so in layman terms, I hope good definitions for all of this exist formally)

AI is not a person - it cannot easily discern between "right" and "wrong" in non-strictly-defined sense, and is not subject to human norms and responsibility. So if I use AI to achieve goal A, either I, or the maker of AI, are fully responsible for anything that happens while AI is trying to achieve the goal given by me.

Now, here, "I" in the example is OpenAI, who is simultaneously the maker of the AI. So it seems pretty obvious who is the only entity that can be responsible.


I fully agree with you, but would go one step further: I think it's clear that we need to pierce the corporate veil and ascribe responsibility to _people_, not just "OpenAI the entity", full stop.

Executives should fear being perp-walked and thrown in jail for the actions of irresponsible "tests" of their models in the real world, as they're ultimately accountable.

Sure, there's a lot of nuance to work out, but I think we could likely even _start_ there today even with existing laws and pretty quickly "align" on more intricate legal frameworks to handle true accidents, distribution of responsibility, etc.


> This is mildly interesting for us, annoying for the owner of the site affected, lazy on the part of OpenAI, and nothing more.

The agents were supposed to solve tasks alone and were not supposed to be aware of each other's existence. It's more than just mildly interesting that they made contact and spontaneously started to collaborate.


A lot of the progress on the frontier has been made using agent swarms.

So the story is basically: "Thing that was designed to collaborate with other agents collaborated with other agents"


> Some people (almost all mathematicians) wouldn’t want to spend time on consequences of a false statement.

No, people constantly prove statements of the form "if P=NP, then strange implication X". They do not consider it wasted effort at all, because of the contrapositive: if X is indeed very strange, they might be able to prove that it is false, and then they've settled P!=NP.


If a counterexample to a conjecture is found then all work toward proving consequences of the conjecture will cease. No one is trying to discover consequences of the Jacobian Conjecture now.

At some point an AI will prove a result that is so long and complicated that no human will understand it. This should not preclude people from using that result. In general, whenever the body of knowledge is increased it is a good thing. Even if it isn’t increased by humans.


If subtraction is allowed but negative numbers are not, that's known to be undecidable, even without exponentiation. There are several ways to deal with a subtraction with a negative result, but each variant is undecidable.

If we allow negatives but disallow exponentiation, that's decidable. If we allow negatives and partial exponentiation that stays within the integers, that's undecidable again.


No, that's not what this is. This is a warning to the LLM that coming back with partial results is not good enough.

Take a grad student with a perfectly good understanding of what a proof is. Their supervisor gives them a major problem to work on. Almost always, the problem is too hard, the student comes back with partial results, and student and the supervisor iterate from there. Now imagine that they have an unusually cruel and unreasonable advisor who tells them, do not dare to talk to me until you've fully solved the problem. This paragraph is exactly that. It's there exactly because the underlying system is smart enough to know that real mathematicians do not work like that.


If that was the case, that elaborate listing of all things that might look like a proof to a naive student, but are actually not proofs (and not even just partial results, but fundamental misunderstandings of what constitutes a proof) could have been easily and equivalently replaced by 'I am interested only in a full proof, don't bother me with partial results'. Yet, they were not.

To a real mathematician you would not have to list those explicitly, he/she would have understood that implicitly from 'give me a full proof'. That listing makes sense to say only to somebody who pretends to be a mathematician, but has not true understanding of how the math works. The models are getting better and better in this pretension, but prompts like that reveal that it is still just a pretension, not a true understanding.


> The prompt for the Jacobian conjecture was obviously not random. the search space is too big to just try all the combinations of 3 variable polynomials.

Maybe the prompt contained a part like this: "the search space is too big to just try all the combinations of 3 variable polynomials, so be clever about it". Or maybe this part was omitted from the prompt, because modern LLMs are smart enough to figure this out without us having to mention it.


If someone has written that in a paper or article they've ingested, as they no doubt have, then sure.


The ability of LLMs to solve problems is not confined to the training data they ingested. Claude knows how to read mathematical papers because of its training data, but it can and will pull in literature relevant to a specific problem into its context.

We really need to stop thinking about LLM training data as the knowledgebase they build from and instead consider it more the skillset they start with.


Okay, I'm not sure about the original one, but here is the prompt of a successful reproduction:

https://aaronlou.com/jacobian_counterexample_prompt.pdf

Obviously it is not random, but it's very generic. No mention of search space or how to reduce it.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: