I like the idea to consider the token cost for the candidate, good.
But why not keep it "on paper", asking how the candidate would use an LLM or agent and why?
Actually using an agent means asking the candidate to deliver something that you must validate. And you then need to evaluate the candidate by looking at the output of the LLM indirectly.
Isn't it better to directly ask the candidate to explain how an LLM could be used (or if/when it should be used) for your use cases?
At the end of the day, it is the same old 'interview coding game' problem, and I suspect it is working even worse now, using LLMs and agents.
It looks to me that most engineers are between the AI entusiasts, not Luddites.
We love to use it, we build tools around it, trying to optimize the whole chain... And so on, as always.
I think your reasoning is more for startups and software houses than for engineers, because if the software cost go to "zero", a single engineer still have something to do. A company can't survive without good margins.
At the end of the day, it is the same old 'interview coding game' problem, and I suspect it is working even worse now, using LLMs and agents.