By Preet Gandhi, NYU
IntroductionIn the era of data science, artificial intelligence is making impossible feats possible. Driverless cars, IBM Watson’s question-answering system, cancer detection, electronic trading, etc. are all made possible through the advanced decision making ability of artificial intelligence. The deep layers of neural networks have a magical ability to recreate the human mind and its functionalities. When humans make decisions, they have the ability to explain their thought process behind it. They can explain the rationale; whether its driven by observation, intuition, experience or logical thinking ability. Basic ML algorithms like decision trees can be explained by following the tree path which led to the decision. But when it comes to complex AI algorithms, the deep layers are often incomprehensible by human intuition and are quite opaque. Data scientists may have trouble explaining why their algorithm gave a decision and the laymen end-user may not simply trust the machine’s predictions without contextual proof and reasoning.
Explainability is a multifaceted topic. It encompasses both individual models and the larger systems that incorporate them. It refers not only to whether the decisions a model outputs are interpretable, but also whether or not the whole process and intention surrounding the model can be properly accounted for. They try have an efficient trade-off between accuracy and explainability along with a great human-computer interface which can help translate the model to understandable representation for the end users.
There need to be three steps which should be fulfilled by the system :
1) Explained the intent behind how the system affects the concerned parties
2) Explain the data sources you use and how you audit outcomes
3) Explain how inputs in a model lead to outputs.
The necessity
Explainability is motivated due to lacking transparency of the black-box approaches, which do not foster trust and acceptance of AI generally and ML specifically. Rising legal and privacy aspects, e.g. with the new European General Data Protection Regulations will make black-box approaches difficult to use in Business, because they often are not able to explain why a machine decision has been made.
IBM Watson shook the world when it won jeopardy beating the best human players. But when it was marketed to hospitals to help the oncology department detect cancer, it failed miserably. Cancer detection is a very difficult and serious topic. The doctors as well as the patients were unable to trust the machine at each stage of consulting and treatment as Watson wouldn’t provide the reasons for its results. Moreover when its results agreed with the doctor’s, it couldn’t provide a diagnosis.
Using AI for military practices is a hotly debated topic. Proponents claim that lethal autonomous weapon systems (LAWS) might cause less collateral damage but despite there being large training data to distinguish between combatants and civilians or targets and non-targets, it is very risky to leave the final decision making power to machines. CIA has 137 AI projects, one of which is the automated AI-enabled drones where the lack of explainability of the AI software’s selection of the targets is controversial. The extent of an explanation currently may be, “There is a 95 percent chance this is what you should do,” but that’s it. When the algorithm can’t describe it’s features that contributed towards identifying a legitimate target, it leaves room open for debate on racism and stereotype issues which bias the model. Moreover in case of a false target, explanations would help diagnose the cause of failure and help improve the model.
New methods in academia
There are two main set of techniques used to develop explainable systems; post-hoc and ante-hoc. Ante-hoc techniques entail baking explainability into a model from the beginning. Post-hoc techniques allow models to be trained normally, with explainability only being incorporated at testing time.
Ante-Hoc Methods:
Reversed Time Attention Model (RETAIN): Researchers at Georgia-Tech developed the RETAIN model to help doctors understand the AI software’s predictions. The patients hospital visits data were sent to two RNN’s both of which had attention mechanism. The attention mechanism helped explain which part the neural network was focusing on and which features helped influence its choice.