Whenever you introduce (or grow) the presence of computer automation you need to consider the human / social interactions. You are responsible for the whole system, not just your software.
It is well understood that human diligence falls dramatically in the face of automation that is perceived to be highly reliable.
There is a tendency for software people to say “it has a 5% error rate, so make sure to carefully check every time”. Even though we know that humans simply won’t check carefully the results of a 95% accurate system.
> There is a tendency for software people to say “it has a 5% error rate, so make sure to carefully check every time”. Even though we know that humans simply won’t check carefully the results of a 95% accurate system.
This should really be framed differently during training. "Assume you have 100 neighbors in your area/house. We expect you to kill 5 innocent neighbors each time you use the system, and don't actually think for yourself. (After using it once every day for two weeks, you will have killed half of your neighbors.)"
Suddenly 5% doesn't sound that small/95% not as good to the laymen.
IMO you should increase the error rate synthetically, adding tests. So you get 10 targets to review, but 4 of them are known to be bad and the system is both checking the user, and also increasing the perceived error rate to encourage diligence.
Brilliant! This is the logic behind the birthday problem where you only need 45 people present to have greater than 90% chance that 2 people share a birthday.
Even rare events become very likely when you repeat it. A 5% wrong chance means half are dead after 10 tries
> Even rare events become very likely when you repeat it.
True, but in my opinion 5 innocent dead out of hundred is not remotely small. For any serious automated decision anything larger than per million is completely unacceptable. (For reference, among Germans the amount of Jews killed by Nationalsocialists, is "only" 2.3‰ [160k vs. 69M] I don't think anyone considers that to be small.)
> A 5% wrong chance means half are dead after 10 tries
Btw., after using it for a year every day, the US population would be 2.52 people. And I think the US military plans to use their expensive systems for much more than a year.
A computer can never be held accountable, therefore a computer must never make a management decision.
We have CEOs designing AI systems to replace them (or at least their workload). The DoD has done nothing to provide accountability for this war crime. It's a scapegoat to move on from, not a problem or justice.
It is well understood that human diligence falls dramatically in the face of automation that is perceived to be highly reliable.
There is a tendency for software people to say “it has a 5% error rate, so make sure to carefully check every time”. Even though we know that humans simply won’t check carefully the results of a 95% accurate system.