Vojtěch Franc presents Optimal Strategies for Reject Option Classifiers

In classification with a reject option, the classifier is allowed in
uncertain cases to abstain from prediction. The classical cost-based
model of a reject option classifier requires the rejection cost to be
defined explicitly. The alternative bounded-improvement model and the
bounded-abstention model avoid the notion of the reject cost. The
bounded-improvement model seeks a classifier with a guaranteed selective
risk and maximal cover. The bounded-abstention model seeks a classifier
with guaranteed cover and minimal selective risk. We prove that despite
their different formulations the three rejection models lead to the same
prediction strategy: the Bayes classifier endowed with a randomized
Bayes selection function. We define the notion of a proper uncertainty
score as a scalar summary of the prediction uncertainty sufficient to
construct the randomized Bayes selection function. We propose two
algorithms to learn the proper uncertainty score from examples for an
arbitrary black-box classifier. We prove that both algorithms provide
Fisher consistent estimates of the proper uncertainty score and
demonstrate their efficiency in different prediction problems, including
classification, ordinal regression, and structured output classification.

Published in Journal of Machine Learning Research
https://jmlr.org/papers/v24/21-0048.html

https://v3s.cvut.cz/results/detail/365904