Elod P Csirmaz’s Blog: How to interpret a neural network?

## 23 July 2019

### How to interpret a neural network?

Machine learning solutions and neural networks are used in more and more areas of our lives. At the same time, since it is difficult to grasp how they function and so they can be seen as black boxes, there is a reluctance in their adoption in areas where we need to remain accountable for business and/or legal reasons. This fact also impacts the trust we all place in such systems.

Because of this, providing tools that can help us understand how these models arrive at a certain decision is an area of active research. The tools range from visualisation tools, tools that identify the most significant input, tools that allow experimenting with the input data to understand the relative importance the model attributes to different signals, and extracting rules'' from neural networks which can be reviewed or even used by human decision-makers.

I wrote a short paper and sample code to demonstrate a simple solution that makes it possible to extract linear rules from a neural network that employs Parametric Rectified Linear Units (PReLUs). This is based on introducing a new force, applied in parallel to backpropagation, that aims to reduce PReLUs into identity functions, which then causes the neural network to collapse into a smaller system of linear functions and inequalities. It works well with a number of toy examples, and also has the promise to help with overfitting.