|
Given a set of features vectors $X={vec{x}_1,..,vec{x}_n}$, binary ground truth data $Y={y_1,..,y_n}$ and continuous prediction $bar{Y} = {bar{y}_1,..,bar{y}_n}in [0,1]$, I want to perform some regression analysis to estimate a function $f(vec{x}_p) approx e_p$ where $e_p = abs(y_p - bar{y_p})$. In practice I want to be able to predict the confidence of a classification based on some external features that do not belong to the classifier itself (that is treated as a black box). An example could be: I have a classifier that predicts whether a fish is a salmon or a carp. Given a set of features such as weight,size,eye color etc, I want to find a measure that tells me how reliable the classifier is based on those features. E.g. I might find out the classifier is unreliable for fishes of size smaller than 30cm. I am not an expert in the field and I am looking for the best way to proceed. |
|
K-fold cross-validation is a decent way to evaluate the accuracy of a classifier. Partition your dataset into K equal sized "folds". For i = 1:K, train the classifier using all folds except fold i, then predict on fold i and record "accuracy". Once you are done you have K separate accuracy measurements to aggregate into some sort of overall "score". You can then repeat this process multiple times to get multiple scores so that you have an idea of mean and variance. The key thing is that you cannot make predictions on the same data that was used to fit the model if you want to get a good idea of how good the classifier is. |
