Deep neural networks are consisted of several individual neurons, which come together in complicated and counterintuitive ways to find solutions to a broad array of challenging activities. This intricacy provides neural networks their capabilities but also earns them their rep as intricate and opaque black boxes.  

Comprehending how deep neural networks operate is crucial for detailing their decisions and facilitating us to develop more capable systems. For example, imagine how tough it is attempting to construct a clock with no comprehension of how individual gears come together. One strategy to comprehending neural networks, both within neuroscience and deep learning, is to look into the role of individual neurons, particularly those which are easy to interpret. 

The investigation into the criticality of singular directions for generalization, leverages a strategy inspired by years of experimental neuroscience – looking into the influence of damage – to decide: how critical are small groups of neurons in deep neural networks? Are more easily interpretable neurons also more critical to the network’s computation? 

The performance impact was quantified with regards to damaging the network through deletion of individual neurons in addition to groups of neurons. The experiments had the outcome of two shocking findings: 

• Even though several prior studies have concentrated on comprehending simply interpretable individual neurons (for example, “cat neurons”, or neurons in the hidden layers of deep networks which are merely active in reaction to imagery of cats, it was discovered that these interpretable neurons are no more critical than confusing neurons with tough to interpret activity. 
• Networks which accurately categorize unseen imagery are more resistant to neuron deletion than networks which can just categorize imagery they have seen prior. To put it in different words, networks which generalize well are much less dependant on singular directions that ones which memorise. 

“Cat neurons” might be more interpretable, but they’re not more critical 

Both within the domains of neuroscience and deep learning, easily interpretable neurons (“selective neurons”) which are just active in reaction to imagery of a singular input category, like dogs, have undergone extensive analysis. Within deep learning, this has had the outcome of an emphasis on cat neurons, sentiment neurons, and parentheses neurons. In neuroscience, Jennifer Aniston neurons, amongst others. But, the comparative criticality of these few very selective neurons in contrast to the majority of neurons which possess low selectivity and more puzzling, hard-to-interpret activity has stayed unknown. 

To assess neuron criticality, it was quantified how network performance on image classification activities alters when a neuron is deleted. If a neuron is very critical, deleting it should be very damaging and considerably decrease network performance, whereas the deletion of a not-so-important neuron should have minimal influence. Neuroscientists consistently execute similar experiments, even though they cannot accomplish the fine-grained accuracy which is required for these experiments and readily available in artificial neural networks. 

Shockingly, it was discovered that there was minimal relationship amongst importance and selectivity. To put it in different words, “cat neurons” were no more critical than confusing neurons. This discovery echoes latest research in neuroscience which illustrated that confusing neurons can in reality be really informative, and indicates that we ought to investigate beyond the most easily interpretable neurons in order to comprehend deep neural networks. 

While cat neurons can be more interpretable, they are no more critical than confusing neurons which profess no overt preference.

Even though interpretable neurons are simpler to comprehend intuitively, they are no more critical than confusing neurons in the grand scheme of things.

Networks that generalize better are more difficult to crack

It is the goal of enterprises to build smart systems and we can only refer to a system or framework as being truly ‘intelligent’ when it can generalise and adapt itself to fresh situations. For instance, an image classification network which can only categorize particular dog imagery that it has observed prior, but not fresh imagery of the same dog, is useless. It is only in the smart categorization of new instances that these systems derive their usefulness. A latest collaborative research paper put out by Google Brain, Berkeley, and DeepMind which was awarded best paper at ICLR 2017 illustrated that deep nets can just memorize each and every image on which they receive training over learning in a human-like manner (comprehending the abstract notion of a dog)

Although, it is typically not obvious if a network has learned a situation which can generalize and adapt to new scenarios or not. Through deletion of progressively bigger and bigger groupings of neurons, it was discovered that networks which generalize efficiently were much more robust with regards to deletions than networks which merely memorized imagery that were prior observed during training. To put it in different words, networks that generalize better are difficult to crack (even though they definitely still be broken.)

By quantifying network robustness in this fashion, we can assess if a network is exploiting unwanted memorization to cheat. Comprehending how networks modify when they go through memorization will facilitate us to develop new networks that memorize less and generalize more.

Neuroscience-inspired analysis

Combined, these discoveries illustrate the capability of leveraging strategies inspired by experimental neuroscience to comprehend neural networks. Leveraging these strategies, it was discovered that highly selective individual neurons are no more critical than non-selective neurons, and that networks that generalize better are much less dependent on individual neurons that ones which merely memorize the training information. These outcomes mean that individual neurons might be much less critical than a first observation may indicate.

Through collaboration to illustrate the role of every neuron, not just ones which are simple-to-interpret, the hope is to better comprehend the inner functionality of neural networks, and crucially, to leverage this understanding to develop smarter and general, functional systems.