In many ways, AIs are alien intelligences.  They make decisions in a “black box” and we can’t really understand why they made the choices they made.  Explainable frameworks usually look like the answer.  They turn AIs in “glass box” models, that reveal some of their inner mystery.  If we can just get our machine learning models talking to us then we can trust those decisions.  But it turns out that some explainable models only give us a false sense of confidence. In this article from MIT Tech Journal researchers conducted tests with AI researchers and found some shocking results:

Sure, the tools sometimes helped people spot missing values in the data. But this usefulness was overshadowed by a tendency to over-trust and misread the visualizations. In some cases, users couldn’t even describe what the visualizations were showing. This led to incorrect assumptions about the data set, the models, and the interpretability tools themselves. And it instilled a false confidence about the tools that made participants more gung-ho about deploying the models, even when they felt something wasn’t quite right. Worryingly, this was true even when the output had been manipulated to show explanations that made no sense.