Acta Univ. Agric. Silvic. Mendelianae Brun. 2013, 61(7), 2269-2275 | DOI: 10.11118/actaun201361072269

Three-way ROC analysis using SAS Software

Juraj Kapasný, Martin Řezáč
Department of Mathematics and Statistics, Faculty of Science, Masaryk University, Kotlářská 2, 611 37 Brno, Czech Republic

The most commonly used measure of model accuracy in medicine with three categories of target variable is the volume under ROC surface (VUS), which is the extension of the area under curve (AUC) for binary models (Le and Lili, 2013). This paper deals primarily with usage of the multinomial logistic regression and the three-way ROC analysis in the financial sector, especially in the credit risk management. Moreover, SAS system is very often used software in the financial sector. Therefore this paper is focused on ways of doing three-way ROC analysis in this statistical software, in particular on estimating the VUS.
We propose an estimate of the VUS based on the confusion matrix, which is compared to estimates based on Mann-Whitney statistic and on empirical distribution functions. We developed three SAS macros based on these approaches for computing the VUS. Furthermore, we developed some logistic models for three-value target variable based on the Loss Given Default (LGD). This was done on real financial data. Results obtained by the SAS macros on these models are presented a discussed in the paper.

Keywords: multinomial logistic regression, LGD, Three-way ROC analysis, ROC surface, VUS, SAS Software

Received: August 28, 2013; Published: December 24, 2013  Show citation

ACS AIP APA ASA Harvard Chicago IEEE ISO690 MLA NLM Turabian Vancouver
Kapasný, J., & Řezáč, M. (2013). Three-way ROC analysis using SAS Software. Acta Universitatis Agriculturae et Silviculturae Mendelianae Brunensis, 61(7), 2269-2275. doi: 10.11118/actaun201361072269
Share...
Download citation
  • BibTeX (.bib)
  • Bookends (.ris)
  • EasyBib (.ris)
  • EndNote (.enw)
  • EndNote 8 (.xml)
  • ISI WoS (.isi)
  • Medlars (.medlars)
  • Mendeley (.ris)
  • MODS (.xml)
  • Papers (.ris)
  • RefWorks (.txt)
  • RefManager (.ris)
  • RIS (.ris)
  • MS Word (.xml)
  • Zotero (.ris)
    • Open full article

    References

    1. BRUIN, J., 2011: Newtest: command to compute new test, http://www.ats.ucla.edu/stat/stata/ado/analysis/. Accessed on August 15, 2013.
    2. CORTES, C., MOHRI, M., 2003: AUC Optimization vs. Error Rate Minimization, http://machinelearning.wustl.edu/mlpapers/paper_files/NIPS2003_AA40.pdf. Accessed on July 26, 2013.
    3. FERRI, C., HERNÁNDEZ-ORALLO, J., SALIDO, M. A., 2003: Volume Under the ROC Surface for Multi-class Problems. Exact Computation and Evaluation of Approximations, http://users.dsic.upv.es/~jorallo/escrits/VUS-More-than-2j.pdf. Accessed on June 24,2013."
    4. HOSMER, D. W. and LEMESHOW, S., 2000: Applied logistic regression. 2nd ed. New York: John Wiley & Sons Inc, 375 p. ISBN 978-0-471-35632-5. Go to original source...
    5. LANDGREBE, T., DUIN, R. P. W., 2006: A simplified extension of the Area under the ROC to the multiclass domain. [online] 26. 7. 2013. Available on: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.123.1455&rep=rep1&type=pdf.
    6. LE K., LILI T., 2013: Estimation of the volume under the ROC surface with three ordinal diagnostic categories. Computational Statistics & Data Analysis, Volume 62, June 2013, Pages 39-51, ISSN 0167-9473. Go to original source...
    7. LI J., FINE J. P., 2008: ROC analysis with multiple classes and multiple tests: methodology and its application in microarray studies. Biostatistics, Volume 9, Issue 3, Pages 566-576, ISSN 1527-2486. DOI: 10.1093/biostatistics/kxm050 Go to original source...
    8. MOSSMAN, D., 1999: Three-way ROCs. Medical Decision Making, Volume 19, Issue 1, ISSN 0272-989X. Go to original source...
    9. ŘEZÁČ, M., 2009: Predictive Model Monitoring - Population Stability Indices. FORUM STATISTICUM SLOVACUM, Volume 5, Issue 1, Pages 79-84. ISSN 1336-7420.
    10. ŘEZÁČ, M. and ŘEZÁČ, F., 2011. How to Measure the Quality of Credit Scoring Models. Finance a úvěr - Czech Journal of Economics and Finance, 61, 5, Pages 486-507. ISSN 0015-1920.
    11. SCURFIELD, B. K., 1996: Multiple/event forced/choice tasks in the of signal detectability. Journal of Mathematical Psychology, 40, pp. 253-269, ISSN 0022-2496. DOI: 10.1006/jmps.1996.0024 Go to original source...
    12. WAEGEMAN, W., DE BAETS, B., BOULLART, L., 2008: ROC analysis in ordinal regression learning. Pattern Recognition Letters, Volume 29, Issue 1, Pages 1-9, ISSN 0167-8655. DOI: 10.1016/j.patrec.2007.07.019 Go to original source...
    13. WAN, S., 2012.: An empirical likelihood confidence interval for the volume under ROC Surface. Statistics and Probability Letters, 82, pp. 1463-1467. ISSN 0167-7152. DOI: 10.1016/j.spl.2012.04.007 Go to original source...

    This is an open access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY NC ND 4.0), which permits non-comercial use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.


    Return to the content