Estimation of AUC for Assessing Its Significance in Classification Models

Abstract

The assessment of the performance of a diagnostic test when test results are measured on continuous scale can be evaluated using the measures of sensitivity and specificity over the range of possible cut-off points for the predictor variable. This is achieved by the use of a receiver operating characteristic (ROC) curve which is a graph of sensitivity against 1-specificity across all possible decision cut-offs values from a diagnostic test result. This curve evaluates the diagnostic ability of tests to discriminate the true state of subjects especially in classification models. These tasks of assessing the predictive accuracy of classification models is always better achieved using a summary measure of accuracy across all possible ranges of cut-off values called the area under the receiver operating characteristic curve (AUC). In this paper, we propose a simple nonparametric method of calculating AUC from predicted probability of positive response involving multiple prediction rules. This method is based on the knowledge of non-parametric Mann-Whitney U statistic. Based on the predicted outcomes and observed outcomes, the performance of diagnostic tests is assessed for the classification models through the AUC calculated from these outcomes. The proposed method when applied on real data, the significance of AUC for the classification models is assessed. The method offers reliable statistical inferences and circumvents the difficulties of deriving the statistical moments of complex summary statistics seen in the parametric method. The proposed method as a non-parametric estimation is recommended for calculating the AUC as it compares favorably with the existing parametric and non-parametric methods.   

Keywords: Cut-off value, ROC, Predicted probability, parametric, non-parametric

DOI: 10.7176/JNSR/9-9-03

Publication date:May 31^st 2019