Magnitude and Uncertainty Pruning Criterion for Neural Networks

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningfagfællebedømt

Neural networks have achieved dramatic improvements in recent years and depict the state-of-the-art methods for many real-world tasks nowadays. One drawback is, however, that many of these models are overparameterized, which makes them both computationally and memory intensive. Furthermore, overparameterization can also lead to undesired overfitting side-effects. Inspired by recently proposed magnitude-based pruning schemes and the Wald test from the field of statistics, we introduce a novel magnitude and uncertainty (MU) pruning criterion that helps to lessen such shortcomings. One important advantage of our MU pruning criterion is that it is scale-invariant, a phenomenon that the magnitude-based pruning criterion suffers from. In addition, we present a 'pseudo bootstrap' scheme, which can efficiently estimate the uncertainty of the weights by using their update information during training. Our experimental evaluation, which is based on various neural network architectures and datasets, shows that our new criterion leads to more compressed models compared to models that are solely based on magnitude-based pruning criteria, with, at the same time, less loss in predictive power.

OriginalsprogEngelsk
Titel2019 IEEE International Conference on Big Data, Big Data
RedaktørerChaitanya Baru, Jun Huan, Latifur Khan, Xiaohua Tony Hu, Ronay Ak, Yuanyuan Tian, Roger Barga, Carlo Zaniolo, Kisung Lee, Yanfang Fanny Ye
ForlagIEEE
Publikationsdato2019
Sider2317-2326
Artikelnummer9005692
ISBN (Elektronisk)9781728108582
DOI
StatusUdgivet - 2019
Begivenhed2019 IEEE International Conference on Big Data, Big Data 2019 - Los Angeles, USA
Varighed: 9 dec. 201912 dec. 2019

Konference

Konference2019 IEEE International Conference on Big Data, Big Data 2019
LandUSA
ByLos Angeles
Periode09/12/201912/12/2019
SponsorAnkura, Baidu, IEEE, IEEE Computer Society, Very
NavnProceedings - 2019 IEEE International Conference on Big Data, Big Data 2019

ID: 241594680