22 September 2014

Shun-ichi Amari is visiting DIKU

Information geometry

The famous Japanese scientist, Shun-ichi Amari, a pioneer within the field of information geometry, will be lecturing at the DIKU PhD course, “Information Geometry in Learning and Optimization” this week. DIKU’s professor, Christian Igel, to whom the work of Amari has been greatly influential, explains the significance of his research.

Shun-ichi Amari

Photo: RIKEN Brain Science Institute

DIKU’s professor Christian Igel explains the significance of the work of Professor Shun-ichi Amari

Christian Igel- We are honoured that Prof. Shun-ichi Amari will visit us for the course on “Information geometry in learning and optimisation”. He pioneered the research field of information geometry, a field which found applications in most major areas of machine learning. Most significantly for the research done here at DIKU, information geometry suggests to follow the so-called ‘natural gradient’ for parameter adaptation of probabilistic models (loosely speaking the gradient in the space of probability distributions) - and this results in many beautiful algorithms.

- Recently, it was discovered that one of my favourite algorithms, the CMA-ES, can also be understood from the viewpoint of information geometry. I regard this as one of the most exciting current research direction within randomised search.

- Today, using gradient-based optimization for training adaptive systems is an obvious choice, perhaps the most popular one. Without getting into the debate about who “invented" it, it is fair to say that Prof. Amari was among the first who discussed the use of gradient descent when training neural networks in his article "A Theory of Adaptive Pattern Classifiers” from 1967.

- While my colleagues and I frequently use algorithms based on the natural gradient, my first contact with Prof. Amari’s work was in a very different context, namely the context of modelling information processing in the brain. For anybody interested in modelling the interactions of neurons at tissue level, Shun-ichi Amari’s papers on pattern formation in neural fields are a must read.

Shun-ishi Amari selected bibliography, or Christian Igel’s reading recommendations

Shun-ichi Amari was among the first who discussed the training of neural network type classifiers by minimising errors by means of gradient descent:

Shun-ichi Amari
A Theory of Adaptive Pattern Classifiers
IEEE Transactions on Electronic Computers, 1967

Furthermore, he did very influential work on the pattern formation in non-linear neural fields, which are spatiotemporal models of populations of neurons:

Shun-ichi Amari
“Dynamics of pattern formation in lateral inhibition type neural fields”
Biological Cybernetics, 27:77–87, 1977.

And, of course, in 2000 he pioneered the field of information geometry:

Shun-ichi Amari, Hiroshi Nagaoka
Methods of Information Geometry
American Mathematical Society, 2000