What is Explainable AI?

On a more social front, we know that people don’t always make great decisions, and we know that people are not always good at justifying the decisions they make. In fact, people disagree with each other all the time!

So, where do we as a society draw the line and say “OK AI, I don’t agree with you, but you have adequately explained how you reached that conclusion, and I trust the decision you have made.”

Because the field is moving so fast, the willingness for someone to trust what an AI system outputs is still quite a personal thing. At Peak, we’ve previously found that, while one person might be happy to accept AI into their workflow without explainability so long as its performance is demonstrably strong (perhaps through an A/B test), another person in the same role might want a full break down of every output, with plots and a plain English summary.

I expect that as society uses AI more, and becomes more accepting of it, the focus of Explainable AI will shift, becoming less “why should a less technical person believe this?” and more “show me that you aren’t being biased or unethical.”

On the above: one understated property of Explainable AI concepts is their ability to find biases in what we currently do without AI.

Explainable AI can act like the old fashioned canary in the coal mine. If an Explainable AI model says, “do not hire this person because they are a woman,” it will usually be because it learned that the underlying processes themselves are biased against women. Now, some companies seem to blame the canary for its silence – but that is another issue altogether!