Algorithms for estimating the partition function of restricted Boltzmann machines: (Extended Abstract)

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningfagfællebedømt

Estimating the normalization constants (partition functions) of energy-based probabilistic models (Markov random fields) with a high accuracy is required for measuring performance, monitoring the training progress of adaptive models, and conducting likelihood ratio tests. We devised a unifying theoretical framework for algorithms for estimating the partition function, including Annealed Importance Sampling (AIS) and Bennett's Acceptance Ratio method (BAR). The unification reveals conceptual similarities of and differences between different approaches and suggests new algorithms. The framework is based on a generalized form of Crooks' equality, which links the expectation over a distribution of samples generated by a transition operator to the expectation over the distribution induced by the reversed operator. Different ways of sampling, such as parallel tempering and path sampling, are covered by the framework. We performed experiments in which we estimated the partition function of restricted Boltzmann machines (RBMs) and Ising models. We found that BAR using parallel tempering worked well with a small number of bridging distributions, while path sampling based AIS performed best with many bridging distributions. The normalization constant is measured w.r.t. a reference distribution, and the choice of this distribution turned out to be very important in our experiments. Overall, BAR gave the best empirical results, outperforming AIS.

OriginalsprogEngelsk
TitelProceedings of the 29th International Joint Conference on Artificial Intelligence, IJCAI 2020
RedaktørerChristian Bessiere
Antal sider5
ForlagInternational Joint Conferences on Artificial Intelligence
Publikationsdato2020
Sider5045-5049
ISBN (Elektronisk)9780999241165
DOI
StatusUdgivet - 2020
Begivenhed29th International Joint Conference on Artificial Intelligence, IJCAI 2020 - Yokohama, Japan
Varighed: 1 jan. 2021 → …

Konference

Konference29th International Joint Conference on Artificial Intelligence, IJCAI 2020
LandJapan
ByYokohama
Periode01/01/2021 → …
SponsorInternational Joint Conferences on Artifical Intelligence (IJCAI)
NavnIJCAI International Joint Conference on Artificial Intelligence
Vol/bind2021-January
ISSN1045-0823

ID: 256580661