Knowledge sharing for population based neural network training

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningfagfællebedømt

Finding good hyper-parameter settings to train neural networks is challenging, as the optimal settings can change during the training phase and also depend on random factors such as weight initialization or random batch sampling. Most state-of-the-art methods for the adaptation of these settings are either static (e.g. learning rate scheduler) or dynamic (e.g ADAM optimizer), but only change some of the hyper-parameters and do not deal with the initialization problem. In this paper, we extend the asynchronous evolutionary algorithm, population based training, which modifies all given hyper-parameters during training and inherits weights. We introduce a novel knowledge distilling scheme. Only the best individuals of the population are allowed to share part of their knowledge about the training data with the whole population. This embraces the idea of randomness between the models, rather than avoiding it, because the resulting diversity of models is important for the population’s evolution. Our experiments on MNIST, fashionMNIST, and EMNIST (MNIST split) with two classic model architectures show significant improvements to convergence and model accuracy compared to the original algorithm. In addition, we conduct experiments on EMNIST (balanced split) employing a ResNet and a WideResNet architecture to include complex architectures and data as well.

TitelKI 2018 : Advances in Artificial Intelligence - 41st German Conference on AI, 2018, Proceedings
RedaktørerAnni-Yasmin Turhan, Frank Trollmann
Antal sider12
ForlagSpringer Verlag,
Publikationsdato1 jan. 2018
ISBN (Trykt)9783030001100
StatusUdgivet - 1 jan. 2018
Eksternt udgivetJa
Begivenhed41st German Conference on Artificial Intelligence, KI 2018 - Berlin, Tyskland
Varighed: 24 sep. 201828 sep. 2018


Konference41st German Conference on Artificial Intelligence, KI 2018
NavnLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Vol/bind11117 LNAI

ID: 223196024