As the silliest example of this: the Calculus of Inductive Constructions (the core calculus implemented in Lean) has a published proof of its soundness (that you cannot derive False in it). That proof can (probably, thanks to Gödel) not be formalized in Lean or any formal system of axioms weaker than Lean.
If what you were trying to do was define what it means for something to be a correct proof, then you must equally admit that there can't be a correct proof of the fact that Lean can't prove False. And if you find that world a happy world to live in while we delegate all concerns about truth to Lean, then I have an infinite hierarchy of monotonically growing bridges to sell you.
Any proof which is correct can be formalized. That is what is means for a proof to be correct.
I’m not sure why you are restricting the scope of our discussion to what _Lean_ can(not) formalize. Even if we do choose to do that for some reason, Lean is not a formal system of fixed axioms; it supports optional additional axioms.
There is no unconditional proof that CIC is sound. Thanks to Gödel, the best you can do is relative consistency proofs, something like "in ZFC with omega (i.e. infinitely) many inaccessible cardinals, you can prove that the theory of Lean is consistent".
Such a relatively consistency proof can be fully formalized in Lean (and in fact, in much weaker systems than Lean).
In principle, a computer can check a nonconstructive proof. I don't know if that's possible in Lean. The interpretation of proofs still assumes the consistency of their axioms.
But hang on. None of this argument is about companies or people using AI being unfair in their practices. It is about companies and people training models and building AI products being unfair in their practice. (By forcing others to choose between adopting unscrupulous practices or dying.)
I don't think I said anything about people using ai being unfair in their practices, the unfair part being that people are forced to adopt products they find unscrupulous. It's hard to argue you find a thing "unscrupulous" (1934 case) when you use it while also liking the outcomes.
> It is about companies and people training models and building AI products being unfair in their practice.
Khan's argument is tied to an "unfair" definition in a 1934 law, not the colloquial usage today about unfair around copyright usage, which is also debated. The "unfair" there is about any activity which leads into a spiral of decreasing public good. The dark patterns of social media are another place we ought to use this law to hold companies and execs accountable. It's "unfair" because others feel compelled to adopt the practice or lose their business. The News Media industry is another example where this has played out. It is arguable that developers feel "unfairly" compelled to adopt Ai or lose their trade. Certainly many in the field have openly expressed this anxiety. While not exactly the same situation as the case (companies in competition with each other), it is developers in competition with each other in the (perceived to be taken by ai) job market. This was the analogy I was making, which is closer to the 1934 case than the "unfair" copyright around Ai that is widely debated.
As the silliest example of this: the Calculus of Inductive Constructions (the core calculus implemented in Lean) has a published proof of its soundness (that you cannot derive False in it). That proof can (probably, thanks to Gödel) not be formalized in Lean or any formal system of axioms weaker than Lean.
If what you were trying to do was define what it means for something to be a correct proof, then you must equally admit that there can't be a correct proof of the fact that Lean can't prove False. And if you find that world a happy world to live in while we delegate all concerns about truth to Lean, then I have an infinite hierarchy of monotonically growing bridges to sell you.