Questions tagged [cross-validation]
Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
3,518 questions
6
votes
1
answer
71
views
How can I evaluate a time‑series forecasting model when I must train on the entire small dataset?
I’m building a Python forecasting pipeline that tries several models:
Holt‑Winters (tuned with Optuna)
ARIMA (via pmdarima.auto_arima)
XGBoost (tuned with Optuna)
...
- 161
2
votes
0
answers
61
views
Do k-folds risk sampling bias and, if so, how do we avoid it?
In cross-validation, $k$-folds are a common way to train, compare and validate models. Often we want to find an optimal set of hyperparameters for our models. There are many ways to probe the ...
- 311
2
votes
1
answer
58
views
Should differential expression analysis be incorporated in cross validation for training machine learning models?
I'm conducting some experiments using TCGA-LUAD clinical and RNA-Seq count data. I'm building machine learning models for survival prediction (Random Survival Forests, Survival Support Vector Machines,...
- 21
2
votes
0
answers
59
views
Cross-validating multi-output models: importance + SHAP
I am currently developing a project that deals with multiple targets which can have different numbers of cardinalities. The idea is to use different ML-models(e.g. Random Forest, SVM, AdaBoost) and ...
0
votes
0
answers
25
views
What is the best way to determine if cross validated R-squared scores are significantly different? [duplicate]
I'm comparing, pairwise, the results of Linear Regression models with transformations applied to one numerical feature and the target. I'm using K folds cross validation scoring with R-squared. The ...
- 1
1
vote
0
answers
56
views
How to choose between ARIMA and ARFIMA?
I am in the position of having a time series data set that I can model well using either a Autoregressive Fractionally Integrated Moving Average (ARFIMA) or an ARIMA model. I'm asking for ways to ...
- 297
4
votes
1
answer
519
views
Should I normalize both train and valdiation sets or only the train set?
I have a question about normalization when merging training and validation sets for cross-validation.
Normally, I normalize using re-scaling (Min-Max Normalization) calculated from the training set ...
1
vote
2
answers
273
views
A proper approach to K-fold cross validation on imbalanced data
What is the proper algorithm for k-fold CV in case of class-balancing (under/over sampling)?
Variant 1:
split data into train and test set
balance classes in the train set
run k-fold CV
Variant 2:
...
- 378
4
votes
1
answer
133
views
When and how can unsupervised preprocessing before splitting data lead to overoptimistic model performance?
Conceptually, I understand that models should be built totally blind to the test set in order to most faithfully estimate performance on future data. However, I'm struggling to understand the extent ...
- 329
0
votes
0
answers
55
views
LASSO and cross validation when dealing with missing data
I want to simulate data with missing values and use them to compare the predictive performance of several machine learning algorithms, including LASSO. All analyses will be performed in R, using the ...
- 563
4
votes
1
answer
89
views
Confused about the utility of nested cross-validation vs k-fold cross-validation
I am using nested cross validation in mlr3 to tune my model's hyperparameters and gauge its out-of-sample performance. Previously, when I was performing regular k-fold CV, my understanding was that ...
1
vote
1
answer
122
views
How to choose and structure a GLM for species richness with non-normal distribution? [closed]
I know my next steps involve using a GLM and selecting the type of GLM based on my response variables (possibly gamma or Poisson regression?).
I also need to standardise explanatory variables to be ...
- 41
0
votes
1
answer
147
views
Comparing AUROCs of binary classifiers across cross-validation folds: alternatives to DeLong
I have two binary classifiers and would like to check whether there is a statistically significant difference between the area under the ROC curve (AUROC). I have reason to opt for AUROC as my ...
2
votes
0
answers
31
views
How can one statistically compare machine learning models based on the results of a cross validation? [duplicate]
It is often recommended that one uses cross fold validation to estimate the generalisation ability of a machine learning model. Most ressources I've found however do not adres what one should do after ...
- 121
0
votes
0
answers
64
views
Time series LASSO K-fold cross validation
This topic has been discussed before but I couldn't find a specific answer.
Here's my approach to forecast QoQ values,
Run the usual LASSO K-fold CV on timeseries data and generate a one-step ahead ...
- 1
0
votes
1
answer
60
views
Data cross validation to predict label from cluster analysis [closed]
My project has the following steps:
Use elbow method to determine the features and number of clusters for kmeans.
Run kmeans on the data (with determined features and n clusters), and gives the ...
- 103
2
votes
1
answer
111
views
When Does a Mediation Model Qualify as SEM Without a Direct IV→DV Path?
I’m trying to understand structural equation modeling (SEM) for hypotheses model and have questions about when to apply SEM. I have three models in mind:
• Model 1: IV → M → DV
• Model 2: IV → M1 →...
- 85
2
votes
1
answer
160
views
How many folds should a unnnested CV have compared to a nested CV
I read in the mlr3 book about nested resampling that:
Nested resampling is a method to compare models and to estimate the generalization
performance of a tuned model, however, this is the performance ...
- 43
1
vote
1
answer
122
views
Huge steps in AUROC plot
I'm building a model for a binary classification task. Because my dataset is pretty small (~86 samples with 68 class 0 and 18 class 1), I'm using a nested k-fold cross validation (5-inner loops and 5-...
- 13
1
vote
0
answers
55
views
How to compare two kappa statistics from the same group of raters, rating the same subjects, but under two different conditions?
Is there a statistical way to compare two kappa statistics from the same group of raters, rating the same subjects, but under two different conditions (low vs. high field strength MRIs)? We can't ...
- 13
3
votes
1
answer
96
views
K-folds cross validation application
We have a small dataset of n=130. Current step is exploring the data looking for anything interesting. Our primary aim is to compare whether using additional variable is helping improve model ...
- 33
4
votes
1
answer
274
views
Simple procedure for feature selection given correlated predictors
I am trying to make a linear regression predictive model between a continuous dependent variable and a set of continuous predictors. I have a large number (~5000) of these predictor variables (...
- 145
1
vote
0
answers
42
views
Calibrated Classifier on Training Data [closed]
If I am using a GridSearchCV to find hyper parameters on a training set; if I were to run a CalibriatedClassifierCV to tune my probabilities, would it suffice to fit the CalibraitedClassifierCV with ...
- 89
0
votes
0
answers
51
views
Calculating Standard Deviation of RMSE of an unsupervised algorithm
If there is an ML model, the standard deviation (SD) of the root mean squared error (RMSE) can be calculated using time series splits by fitting the model on different training sets and evaluating it ...
- 329
1
vote
1
answer
87
views
Comparing two cross validation methods for hyperparameter tuning
For cross validation of hyperparameters, I have a question about which approach is generally considered better in the context of running regularized regression (specifically elastic net l1, l2 ...
- 111
6
votes
1
answer
133
views
Evaluating a model in a small sample using a test set: bootstrap vs. LOOCV
The thread Evaluating a classifier with small samples considers the problem in its title. Specifically, the question is about splitting off the test set from the rest of the data many times instead of ...
- 71.5k
6
votes
2
answers
203
views
Evaluating classifier with small samples
I'm trying to evaluate two classifiers splitting the sample into the training and tests samples with 50-50 split. The classifiers are fitted and tuned with K-fold CV on the training sample. The ...
- 487
0
votes
0
answers
34
views
Nested linear model comparison and regression parameter testing in LOOCV setting?
How do I obtain a reasonable parameter estimate (regression beta) for the single predictor of interest in a multiple regression model and appropriate standard errors for this estimate using holdout ...
- 312
1
vote
0
answers
57
views
The use of cross-validation and a hold-out set
I've been thinking about the use of cross-validation and hold-out sets and I don't really see the use of a randomly selected hold-out test set. I have to say, though, that when the hold-out is not ...
- 11
1
vote
1
answer
71
views
Is it okay to select any of the surrogate models in nested cv?
Let's say I pick any of the winning surrogate models in my nested cv (in theory if you do k outer folds you could have k surrogate models) to simplify things, lets say I pick the first model and just ...
- 185
0
votes
0
answers
78
views
Interpreting Nested CV Results When Selected Model Didn't Win All Outer Folds
In nested cross validation, I'm seeing an interesting scenario that I'd like to understand better:
Using 4-fold outer CV, my model selection process chose Model A overall (it performed best on average ...
- 185
0
votes
0
answers
88
views
Use cross validation to determine number of factors in factor analysis: why the case is not simply that more factors get larger likelihood?
Consider a factor analysis model
\begin{equation*}
\begin{array}{cccccccccc}
X &=& \mu&+& L&\cdot& f & + &u \\
p\times 1 & & p\times 1 &&p\times k& ...
user398751
0
votes
0
answers
58
views
Can't understand the evaluation approach used in this paper
In this paper, two deep learning models where proposed: Hybrid-AttUnet++ and EH-AttUnet++. The first model, Hybrid-AttUnet++, is simply a modified U-net model, and the second model is an ensemble ...
- 1
4
votes
1
answer
146
views
Use cross validation to select ridge regression parameter $k$: What if mean of $\mathbf x_i$ and $y_i$ might be non-zero on test/training set?
Consider a regression model $$ Y= X\beta+ u. \tag{$\star$} $$
$Y$ is a column vector with length $n$ containing $n$ observations.
$X$ is a $n\times p$ matrix with each row corresponding to a ...
user398751
0
votes
0
answers
36
views
Error when using stratified samples with MERT in LongituRF package in R
I'm using the LongituRF package in R to fit a MERT (Mixed effects regression trees) model to my data. While I have no issues ...
- 399
0
votes
0
answers
54
views
Classification strategies for small biomedical dataset with imbalanced classes
I have spectroscopy data measured from 10 different porcine. The goal is to analyse three different tissue types. However, not all tissues were measured from each porcine. The total numbers are Fat: 3,...
- 119
2
votes
1
answer
187
views
Multinomial logistic regression, Ridge regression
I am currently working with a dataset that includes sociodemographic information about each student in a class (X variables) and information about whom each student votes for as class speaker (Y ...
- 51
0
votes
0
answers
43
views
Safe to break up k-fold cross validation grid search into separate chunks?
I'm performing gradient boosting machine modeling on a large dataset (700k+ records) with several hundred variables on a work laptop with limited memory. I'm coding in R v2022.02.2.
I've found running ...
- 6,644
1
vote
0
answers
34
views
Do I have to get another separate test set that is independent of the dataset I used in cross-validation?
What I'm doing
I am making an undergraduate thesis about audio classification using SVM. My goal is to identify if adding Feature X to the feature matrix could improve the performance of the ...
- 11
0
votes
0
answers
26
views
Does model retrain frequency in time series CV have to match production retrain frequency?
Lets assume that we retrain the model every year in production and we have accumulating 50 years of data. If using a time series CV (e.g TimeSeriesSplit in sklearn) for hyperparams recalibration at ...
- 121
1
vote
2
answers
284
views
GAM Leave one out cross validation (LOOCV) for biggish models
I have fitted a relatively complex/large generalized additive model for prediction purposes but would like to assess its predictive power/cross-validate it. Due to variability in observed data and the ...
- 362
3
votes
1
answer
372
views
What should the objective be when tuning hyperparameters to minimize overfitting?
I'm working on a classification problem with ~90k data rows and 12 features. I'm trying to tune the hyperparamters of an XGBoost model to minimize the overfitting. I use ROC_AUC as the metric to ...
0
votes
0
answers
76
views
How do I find correlation between variables in a time series across multiple days?
I have data for each day, with a date/time, event, and when a secondary event gets triggered.
...
- 101
1
vote
1
answer
104
views
Lasso and cross validation: model selection
Apologies for cross-posting
I am starting to use Lasso and cross validation for model selection to explain a dependent variable using linear models, but I can not understand why all p-values ...
5
votes
3
answers
277
views
Cross-validated bandwidth for the derivative of the function with local quadratic estimation
I am trying to estimate non parametrically the first order derivative of a function g(x). I am estimating $g(x)$ using a local polynomial (quadratic) procedure. I know how to compute the leave-one-out ...
- 239
0
votes
0
answers
45
views
Youtube Spam Classifier - Different Methods yielding the same accuracy (94%)
(CONTEXT)
I'm currently doing a report project at my university to build a classifer model that classifies a comment as spam or ham (non-spam) using this data set, and then submit a prediction csv ...
- 391
0
votes
1
answer
225
views
What’s the appropriate statistical test to compare ML model performance over CV folds?
I’m comparing the performance of 10 ML models across 15-fold cross-validation, using metrics like MSE. Each model’s performance is ranked per fold, and I want to determine if there are significant ...
- 921
1
vote
0
answers
201
views
Error in fitting Zero inflated negative binomial in Python using cross validation
I want to assess predictive power of zero-inflated negative binomial model in Python. My steps are listed as below:
Regarding 5-folds cross-validation:
Fit multiple Zero-Inflated Negative Binomial (...
- 137
16
votes
2
answers
841
views
Advantages of information criteria over cross-validation
I understand AIC is asymptotically equivalent to leave-one-out cross-validation and that BIC has a similar asymptotic equivalence to leave-k-out cross-validation. My question is, other than ...
- 271
0
votes
1
answer
71
views
Separate Test Set for Cross-Validation for Small Sample (n=140)
I’m working on a survival analysis model with a small internal dataset (n=140). An outside researcher suggests splitting the dataset into train/val and setting aside a separate test set (e.g., ~10%, ...
- 1