2
$\begingroup$

Can AUC be used for model selection, and how can the excessive number of features/parameters be penalized in this case?

In frequentist framework we have various model selection criteria, like AIC, BIC, etc., which penalize the excessive number of features - balancing it with likelihood increase (in this sense, BIC is a frequentist criterion.)

In Bayesian framework we select the model with the maximum evidence, where the excessive number of features is penalized via the prior.

In classification tasks, yet another approach is using n-fold cross-validation, where we can compare AUC (or some other score) for different models. This approach is appealing since AUC (or a relevant score) is easier to interpret than information criteria or Bayesian evidence, and it seems (superficially) independent on out-of-blue choices (like Bayesian prior or specific information criterion.)

However, it seems that adding extra features might increase the AUC and must be penalized (which would require introduction of of ad-hoc penalty.) The question is how such a penalty can be introduced.

Improve this question
$\endgroup$
12
  • 1
    $\begingroup$ Setting aside issues with using AUC in particular as the measure of performance, isn’t this what out-of-sample testing does? $\endgroup$ Commented Mar 19 at 9:57
  • $\begingroup$ @Dave I am not familiar with this approach, so I will look it up. Feel free to expand it into an answer. $\endgroup$ Commented Mar 19 at 10:09
  • $\begingroup$ @Dave or do you mean just testing on training vs. validation datasets? In this case it is not clear how this solves the problem - unless you imply that AUC would automatically decrease, if too many features are included. I used to thinks so... but I have encountered a situation where this is manifestly not the case (while Bayesian evidence gives a clear peak.) $\endgroup$ Commented Mar 19 at 10:12
  • 1
    $\begingroup$ @Dave A quick simulation I ran shows terrible AUC values when I add a huge number of features - this is a commonplace behavior, but is there a statement that this is always true (i.e., true for any type of model)? My AUC is everywhere above 0.65, and I use my own function to calculate it (but I tested in the past that its results are identical with that of the relevant sklearn functions.) $\endgroup$ Commented Mar 19 at 10:58
  • 2
    $\begingroup$ That can be overridden (I think penalty = None does it). Earlier versions did not allow for the penalty to be ditched, however. You had to hack around. There should be old Stack Overflow posts about it. Anyway, the penalty probably influences the behavior you’re seeing. I think you should be able to drive the AUC pretty low if you ditch the penalty. $\endgroup$ Commented Mar 19 at 11:28

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.