Format

Send to

Choose Destination
Knowl Inf Syst. 2018 Jan;54(1):151-170. doi: 10.1007/s10115-017-1133-2. Epub 2017 Nov 17.

Binary Classifier Calibration Using an Ensemble of Piecewise Linear Regression Models.

Author information

1
Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA.
2
Department of Biomedical Informatics, Harvard Medical School, MA, USA.
3
Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, USA.

Abstract

In this paper we present a new non-parametric calibration method called ensemble of near isotonic regression (ENIR). The method can be considered as an extension of BBQ (Pakdaman Naeini, Cooper and Hauskrecht, 2015b), a recently proposed calibration method, as well as the commonly used calibration method based on isotonic regression (IsoRegC) (Zadrozny and Elkan, 2002). ENIR is designed to address the key limitation of IsoRegC which is the monotonicity assumption of the predictions. Similar to BBQ, the method post-processes the output of a binary classifier to obtain calibrated probabilities. Thus it can be used with many existing classification models to generate accurate probabilistic predictions. We demonstrate the performance of ENIR on synthetic and real datasets for commonly applied binary classification models. Experimental results show that the method outperforms several common binary classifier calibration methods. In particular, on the real data we evaluated, ENIR commonly performs statistically significantly better than the other methods, and never worse. It is able to improve the calibration power of classifiers, while retaining their discrimination power. The method is also computationally tractable for large scale datasets, as it is O(N logN) time, where N is the number of samples.

KEYWORDS:

ELiTE; ENIR; accurate probability; classifier calibration; ensemble of inear trend estimation; ensemble of near isotonic regression

Supplemental Content

Full text links

Icon for PubMed Central
Loading ...
Support Center