MINT: Deep Network Compression via Mutual Information-based Neuron Trimming
Published in ICPR, 2021
Summary: This project introduces the notion of using conditional mutual informationas a measure of dependence between neurons. The method, titled MINT. focuses on passing through a majority of information while retaining a small percentage of neurons between layers. Using only a single train-prune-retrain step, MINT is extremely competitive with commonly used DNN pruning baselines.
Abstract: Most approaches to deep neural network compression via pruning either evaluate a filter’s importance using its weights or optimize an alternative objective function with sparsity constraints. While these methods offer a useful way to approximate contributions from similar filters, they often either ignore the dependency between layers or solve a more difficult optimization objective than standard cross-entropy. Our method, Mutual Information-based Neuron Trimming (MINT), approaches deep compression via pruning by enforcing sparsity based on the strength of the relationship between filters of adjacent layers, across every pair of layers. The relationship is calculated using conditional geometric mutual information which evaluates the amount of similar information exchanged between the filters using a graph-based criterion. When pruning a network, we ensure that retained filters contribute the majority of the information towards succeeding layers which ensures high performance. Our novel approach outperforms existing state-of-the-art compression-via-pruning methods on the standard benchmarks for this task: MNIST, CIFAR-10, and ILSVRC2012, across a variety of network architectures. In addition, we discuss our observations of a common denominator between our pruning methodology’s response to adversarial attacks and calibration statistics when compared to the original network.
Recommended citation: Ganesh, M.R., Corso, J.J. and Sekeh, S.Y., ‘‘MINT: Deep Network Compression via Mutual Information-based Neuron Trimming’’, ICPR 2020.