Generalized measures of association for ranked data with an application to prediction accuracy

A common practice in cross validation research in the behavioral sciences is to employ either the product moment correlation or a simple tabulation of first-choice “hits” for measuring the accuracy with which various preference models predict subjects’ responses to a holdout sample of choice objects. We propose a nonparametric approach for summarizing the accuracy of predicted rankings across a set of holdout-sample options. The methods that we develop contain a novel way to deal with ties and an approach to the different weighting of rank positions.