Empirical Assessment of Nonconformity Score Functions for Classifiers in Conformal Prediction
  IJCTT-book-cover
 		          
 
© 2024 by IJCTT Journal
Volume-72 Issue-7
Year of Publication : 2024
Authors : Bhargava Kumar, Tejaswini Kumar, Swapna Nadakuditi
DOI :  10.14445/22312803/IJCTT-V72I7P114

How to Cite?

Bhargava Kumar, Tejaswini Kumar, Swapna Nadakuditi, "Empirical Assessment of Nonconformity Score Functions for Classifiers in Conformal Prediction," International Journal of Computer Trends and Technology, vol. 72, no. 7, pp.108-112, 2024. Crossref, https://doi.org/10.14445/22312803/IJCTT-V72I7P114

Abstract
Conformal prediction provides statistically guaranteed confidence measures for any machine learning model. This study investigates the effectiveness of three non-conformity score functions, namely Adaptive Prediction Sets (APS), Regularized Adaptive Prediction Sets (RAPS), and Sorted Adaptive Prediction Sets (SAPS), for sentiment analysis tasks. Expanding on past research that demonstrated the superiority of SAPS in classification tasks for image data, this study assesses whether this superiority extends to other domains, such as sentiment analysis. The study aims to evaluate these non-conformity score functions based on coverage and set sizes. The researchers conducted extensive experiments on a sentiment classification task using the GoEmotions dataset to gain insights into the versatility of SAPS and compared its performance with APS and RAPS. By examining the effectiveness of these non-conformity score functions, this study contributes to the understanding of the practicality of conformal prediction methods in real-world machine learning tasks beyond image classification.

Keywords
Adaptive Prediction Sets (APS), Conformal prediction, Non-Conformity Score Functions, Regularized Adaptive Prediction Sets (RAPS), Sorted Adaptive Prediction Sets (SAPS).

Reference

[1] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer, Algorithmic Learning in a Random World, Springer, pp. 1-476, 2022.
[Google Scholar] [Publisher Link]
[2] Jonathan Alvarsson et al., “Predicting with Confidence: Using Conformal Prediction in Drug Discovery,” Journal of Pharmaceutical Sciences, vol. 110, no. 1, pp. 42–49, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Margaux Zaffran et al., “Adaptive Conformal Predictions for Time Series,” Proceedings of the 39th International Conference on Machine Learning, pp. 25834-25866, 2022.
[Google Scholar] [Publisher Link]
[4] Xianghao Zhan et al., “An Electronic Nose-based Assistive Diagnostic Prototype for Lung Cancer Detection with Conformal Prediction,” Measurement, vol. 158, p. 107588, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[5] Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès, “Classification with Valid and Adaptive Coverage,” arXiv, 2020.
[CrossRef] [Publisher Link]
[6] Anastasios Angelopoulos et al., “Uncertainty Sets for Image Classifiers using Conformal Prediction,” arXiv, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[7] Jianguo Huang et al., “Conformal Prediction for Deep Classifier via Label Ranking,” arXiv, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[8] Dorottya Demszky et al., “GoEmotions: A Dataset of Fine-Grained Emotions,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4040-4054, 2020.
[CrossRef] [Publisher Link]
[9] Jesse C. Cresswell et al., “Conformal Prediction Sets Improve Human Decision Making,” arXiv, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[10] Yinhan Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[11] Hugging Face, SamLowe/roberta-base-go_emotions, 2023. [Online]. Available: https://huggingface.co/SamLowe/roberta-base-go_emotions
[12] Hongxin Wei, and Jianguo Huang, “TorchCP: A Library for Conformal Prediction based on PyTorch,” arXiv, 2024.
[CrossRef] [Google Scholar] [Publisher Link]