Artificial Synaptic Pruning
A Fall 2020 study of synaptic pruning in neural networks, where deleting 80–95% of the weights improved test performance.
What it is
Artificial Synaptic Pruning is a Fall 2020 experiment in deleting most of a neural network’s weights, the way the brain removes under-stimulated synapses, to see whether the network gets better. The obvious expectation is that it gets worse. It was my project for COMS E6998, Computation and the Brain, a Columbia graduate seminar I took as an undergraduate.
Experiment design
The design is a 2×2×2 factorial: Adam or RMSProp, 64 or 128 hidden units, and entropy or R² as the metric for choosing which weights to prune. I pruned autoencoders on Fashion-MNIST and a convolutional network on CIFAR-10, removing 5% to 95% of the weights in 5% steps and retraining for five epochs after each pass.
Findings
Pruning up to 80% of the weights didn’t help. Pruning between 80% and 95% improved test performance, so the heavily pruned networks generalized better than the full ones, which fits pruning reducing overfitting. It’s one sweep on two datasets, so I wouldn’t read much past them.