Revisiting Softmax for Uncertainty Approximation in Text Classification
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
Revisiting Softmax for Uncertainty Approximation in Text Classification. / Holm, Andreas Nugaard; Wright, Dustin; Augenstein, Isabelle.
In: Information (Switzerland), Vol. 14, No. 7, 420, 2023.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - Revisiting Softmax for Uncertainty Approximation in Text Classification
AU - Holm, Andreas Nugaard
AU - Wright, Dustin
AU - Augenstein, Isabelle
N1 - Publisher Copyright: © 2023 by the authors.
PY - 2023
Y1 - 2023
N2 - Uncertainty approximation in text classification is an important area with applications in domain adaptation and interpretability. One of the most widely used uncertainty approximation methods is Monte Carlo (MC) dropout, which is computationally expensive as it requires multiple forward passes through the model. A cheaper alternative is to simply use a softmax based on a single forward pass without dropout to estimate model uncertainty. However, prior work has indicated that these predictions tend to be overconfident. In this paper, we perform a thorough empirical analysis of these methods on five datasets with two base neural architectures in order to identify the trade-offs between the two. We compare both softmax and an efficient version of MC dropout on their uncertainty approximations and downstream text classification performance, while weighing their runtime (cost) against performance (benefit). We find that, while MC dropout produces the best uncertainty approximations, using a simple softmax leads to competitive, and in some cases better, uncertainty estimation for text classification at a much lower computational cost, suggesting that softmax can in fact be a sufficient uncertainty estimate when computational resources are a concern.
AB - Uncertainty approximation in text classification is an important area with applications in domain adaptation and interpretability. One of the most widely used uncertainty approximation methods is Monte Carlo (MC) dropout, which is computationally expensive as it requires multiple forward passes through the model. A cheaper alternative is to simply use a softmax based on a single forward pass without dropout to estimate model uncertainty. However, prior work has indicated that these predictions tend to be overconfident. In this paper, we perform a thorough empirical analysis of these methods on five datasets with two base neural architectures in order to identify the trade-offs between the two. We compare both softmax and an efficient version of MC dropout on their uncertainty approximations and downstream text classification performance, while weighing their runtime (cost) against performance (benefit). We find that, while MC dropout produces the best uncertainty approximations, using a simple softmax leads to competitive, and in some cases better, uncertainty estimation for text classification at a much lower computational cost, suggesting that softmax can in fact be a sufficient uncertainty estimate when computational resources are a concern.
KW - efficiency
KW - text classification
KW - uncertainty quantification
UR - http://www.scopus.com/inward/record.url?scp=85166384773&partnerID=8YFLogxK
U2 - 10.3390/info14070420
DO - 10.3390/info14070420
M3 - Journal article
AN - SCOPUS:85166384773
VL - 14
JO - Information (Switzerland)
JF - Information (Switzerland)
SN - 2078-2489
IS - 7
M1 - 420
ER -
ID: 364498618