The model will have to convince the human that it's making the right kind of progress. That will necessarily become part of the improvement loop - either implicitly (human trusts RSI) or explicitly (human gatekeeps every major decision).