Calibrating Structured Output Predictors for Natural Language Processing
Abhyuday Jagannatha, Hong Yu
Machine Learning for NLP Long Paper
Session 4A: Jul 6
(17:00-18:00 GMT)
Session 5A: Jul 6
(20:00-21:00 GMT)
Abstract:
We address the problem of calibrating prediction confidence for output entities of interest in natural language processing (NLP) applications. It is important that NLP applications such as named entity recognition and question answering produce calibrated confidence scores for their predictions, especially if the applications are to be deployed in a safety-critical domain such as healthcare. However the output space of such structured prediction models are often too large to directly adapt binary or multi-class calibration methods. In this study, we propose a general calibration scheme for output entities of interest in neural network based structured prediction models. Our proposed method can be used with any binary class calibration scheme and a neural network model. Additionally, we show that our calibration method can also be used as an uncertainty-aware, entity-specific decoding step to improve the performance of the underlying model at no additional training cost or data requirements. We show that our method outperforms current calibration techniques for Named Entity Recognition, Part-of-speech tagging and Question Answering systems. We also observe an improvement in model performance from our decoding step across several tasks and benchmark datasets. Our method improves the calibration and model performance on out-of-domain test scenarios as well.
You can open the
pre-recorded video
in a separate window.
NOTE: The SlidesLive video may display a random order of the authors.
The correct author list is shown at the top of this webpage.
Similar Papers
Posterior Calibrated Training on Sentence Classification Tasks
Taehee Jung, Dongyeop Kang, Hua Cheng, Lucas Mentch, Thomas Schaaf,

On the Inference Calibration of Neural Machine Translation
Shuo Wang, Zhaopeng Tu, Shuming Shi, Yang Liu,

Learning Constraints for Structured Prediction Using Rectifier Networks
Xingyuan Pan, Maitrey Mehta, Vivek Srikumar,

Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Ryuto Konno, Kentaro Inui,
