3
$\begingroup$

I have a question concerning a large dataset with 94 observation and 15000 variables. For data mining models (boosting, trees, neural networks...) this number of variables are too much and I have to select a subset of them.

Is lasso (or adaptive lasso) a possibility? If not, what is the right way for selecting variables in a short time?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ a bunch of people are going to hop in this thread in a moment and tell you all the different ways this could go wrong. They're right; there are potential problems. But in practice, this is a great way to get things started, and is used all the time. $\endgroup$ Commented Nov 20, 2023 at 17:19
  • 2
    $\begingroup$ @JohnMadden: I hear and obey. $\endgroup$ Commented Nov 20, 2023 at 18:49
  • 1
    $\begingroup$ Welcome to Cross Validated! Let me guess: a manager wants you to "data science" your way out of a mess of data with no direction and only a tiny amount of data, right? $\endgroup$ Commented Nov 20, 2023 at 19:03

1 Answer 1

Reset to default
1
$\begingroup$

94 observations is simply very little, and 15,000 variables is simply very much. The absolutely best approach would be to use your domain knowledge to reduce the number of features drastically before doing anything else.

Failing that, yes, the lasso is a possibility... as in "when you are falling off a 10,000 foot cliff, then having an umbrella to break your fall is better than not having an umbrella, but seriously, you should not be asking about an umbrella at this point in time" kind of way.

With your data, you are almost certain to fit noise. Pretty much regardless of what you do. Statistics can't conjure information out of noise, unless the signal is enormously strong (and if it were, you would not be running the entire analysis, would you?).

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ +1 I would be curious about how stable the set of selected variables is upon bootstrapping the selection process. With the particular numbers in this case, I am pessimistic. $\endgroup$ Commented Nov 20, 2023 at 19:02
  • 1
    $\begingroup$ @Dave: bootstrapping this and seeing how the selected variables change is an excellent idea. I concur in your pessimism. $\endgroup$ Commented Nov 20, 2023 at 19:03
  • 1
    $\begingroup$ It amazes me how little attention selection stability is given in many machine learning circles, even in circles that are decent in traditional statistics. A good economist wouldn't just accept a point estimate. At the very least, she would want a p-value or regression t-stat, perhaps even a confidence or credible interval, yet such practitioners seem content to take feature importance point estimates as lacking uncertainty. Harrell's "bootstrap the feature importance values" might not be the whole story, but at least there is some concern for uncertainty! (It's not just economics.) Amazing... $\endgroup$ Commented Nov 20, 2023 at 19:08
  • $\begingroup$ @Dave: sometimes, when I'm in a cynical mood, I put this down to how people have a hard time dealing with the uncertainty in their model. How do you write a discussion of your model in the paper when a sensitivity analysis shows you that the selected model is essentially random? How do you present your analysis to your boss when you know that with a different RNG seed, your slides would be looking completely different? $\endgroup$ Commented Nov 20, 2023 at 19:17
  • $\begingroup$ I feel like there's a word for that kind of work. (I'm glad to have learned from people in school and on here who have guided me against that.) $\endgroup$ Commented Jan 5, 2024 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.