3
$\begingroup$

I want to do a regression analysis after a GMM. I have a dependent variable with three categories (classes), which differ in their posterior probabilities. That's why I included the posterior probabilities as weights in my multinomial logistic regression. I'm using multiple imputation to account for missing data in my predictor variables. Now I want to include a quadratic term for one of the predictors and am unsure how I can compare the model fit of the model without and the model with the quadratic term? As far as I know using the AIC is not possible in weighted models.

That's my code for the multinom:

models_gew_cr_quad <- list()

for(i in 1:imp_cr$m){  
  dat_i <- complete(imp_cr, i)
  

  dat_long_cr <- dat_i %>%
    pivot_longer(cols = starts_with("CPROB"), names_to = "class", values_to = "weight") %>%
    mutate(class = factor(class, levels = c("CPROB1","CPROB2","CPROB3"), labels = c("1","2","3")))
  
  
  dat_long_cr$class <- factor(dat_long_cr$class, levels = c("2","1","3"))
  
  models_gew_cr_quad[[i]] <- multinom(class ~ x1 + x2 + x3 + x4 + x5 + x6 + x7,                              
                                 data = data,
                                 weights = weight, trace = FALSE)
}
Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

Your complex model makes me assume you have a lot of data. (If you don't, you might want to revisit that complexity in the first place.) In this case, you could assess different candidate models by doing cross-validation: hold out part of your data, fit the model on the rest, predict for the holdout set, assess the prediction, rinse and repeat by cycling through your data so each observation is once in the holdout set.

Of course, you should use an appropriate evaluation setup. Don't use hard classifications and accuracy: Why is accuracy not the best measure for assessing classification models? Instead, use probabilistic predictions and assess these using proper scoring rules, like the Brier or the log score: Why is LogLoss preferred over other proper scoring rules?

The proof of the pudding is in the eating, and I personally would say that the proof of a model is in the predicting. But then again, that may be the forecaster in me speaking. Breiman (2001) might well disagree.

Improve this answer
$\endgroup$
2
$\begingroup$

I'm not sure that you can't use AIC for weighted models. I hadn't heard of this before and Googling didn't reveal any papers on this. Where did you hear this?

However, assuming you can't use AIC (or don't want to) you can use a variety of graphs to help you choose e.g.

  • A parallel box plot of the residuals from the two (or more) models.
  • A quantile quantile plot of those residuals
  • A Tukey mean-difference plot (aka Bland-Altman plot)

then you can decide whether the improved fit (if any) is worth the added complexity.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the answer! I remember one class in University where it was said that weighted models distort the likelihood, so the deviance has to be used as a proxy and the AIC is therefore only approximative. But as I also don't have any papers to back them up, this might not be accurate information. $\endgroup$ Commented Nov 19 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.