Explainability

Explainability is the ability to say why a model produced a particular output, in terms a person can check. It is demanded for three different reasons that are often confused. A regulator or a customer wants a defensible account of a decision that affected someone. An engineer wants to debug a wrong answer. A user wants enough context to decide whether to trust the output in front of them. Those three audiences need different artifacts, and a method that satisfies one can be useless to another. The available techniques answer weaker questions than people assume. Feature-attribution methods report which inputs the output was most sensitive to, which is a statement about the model and not about the world, so a highly attributed feature may be a proxy rather than a cause. Example-based methods show the nearest training cases, which is often more persuasive to a non-specialist than a chart of weights. For language models, a generated explanation of reasoning is text produced by the same process that produced the answer, so it can be fluent, plausible and unrelated to the actual computation, which makes it useful for review and unsafe as evidence. Two practical consequences follow. Where an explanation must be defensible, the reliable path is often a simpler model, or a hybrid where a transparent rule makes the consequential decision and the complex model handles ranking or triage. And where the requirement is really traceability rather than interpretation, the answer is engineering rather than mathematics: record the model version, the inputs, the retrieved context and the output, so the decision can be reconstructed exactly. That record answers most audit questions, and unlike an attribution chart it does not require anybody to believe an interpretation.