Knowledge sharing for population based neural network training
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Finding good hyper-parameter settings to train neural networks is challenging, as the optimal settings can change during the training phase and also depend on random factors such as weight initialization or random batch sampling. Most state-of-the-art methods for the adaptation of these settings are either static (e.g. learning rate scheduler) or dynamic (e.g ADAM optimizer), but only change some of the hyper-parameters and do not deal with the initialization problem. In this paper, we extend the asynchronous evolutionary algorithm, population based training, which modifies all given hyper-parameters during training and inherits weights. We introduce a novel knowledge distilling scheme. Only the best individuals of the population are allowed to share part of their knowledge about the training data with the whole population. This embraces the idea of randomness between the models, rather than avoiding it, because the resulting diversity of models is important for the population’s evolution. Our experiments on MNIST, fashionMNIST, and EMNIST (MNIST split) with two classic model architectures show significant improvements to convergence and model accuracy compared to the original algorithm. In addition, we conduct experiments on EMNIST (balanced split) employing a ResNet and a WideResNet architecture to include complex architectures and data as well.
Original language | English |
---|---|
Title of host publication | KI 2018 : Advances in Artificial Intelligence - 41st German Conference on AI, 2018, Proceedings |
Editors | Anni-Yasmin Turhan, Frank Trollmann |
Number of pages | 12 |
Publisher | Springer Verlag, |
Publication date | 1 Jan 2018 |
Pages | 258-269 |
ISBN (Print) | 9783030001100 |
DOIs | |
Publication status | Published - 1 Jan 2018 |
Externally published | Yes |
Event | 41st German Conference on Artificial Intelligence, KI 2018 - Berlin, Germany Duration: 24 Sep 2018 → 28 Sep 2018 |
Conference
Conference | 41st German Conference on Artificial Intelligence, KI 2018 |
---|---|
Land | Germany |
By | Berlin |
Periode | 24/09/2018 → 28/09/2018 |
Series | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
---|---|
Volume | 11117 LNAI |
ISSN | 0302-9743 |
- Asynchronous evolutionary algorithms, Hyper-parameter optimization, Population based training
Research areas
ID: 223196024