Questions tagged [generalization-error]
The generalization-error tag has no summary.
14 questions
1
vote
1
answer
136
views
What is the current consensus on "using test set as training set, post testing"? [duplicate]
This question is inspired by a blog post by https://www.argmin.net/p/in-defense-of-typing-monkeys and several rumors I've heard from other people who works in machine learning.
The gist of it is that ...
1
vote
1
answer
93
views
Two questions about the VC theory (on the generalization error bound)
In Andrews Ng's machine learning notes (https://cs229.stanford.edu/main_notes.pdf), he introduced the following bound for the difference between generalization error and training error (see the ...
- 3,090
2
votes
0
answers
176
views
How to compare generalization errors of nested predictive models?
Suppose I want to compare two nested predictive models, for example a Lasso regression based predictive algorithm with and without a certain predictor Z. In the first predictive algorithm, Z would not ...
- 149
3
votes
1
answer
203
views
Difference between generalisation error (Vapnik risk) and frequentist (statistical) risk
I'm reading these lecture notes:
http://www.iro.umontreal.ca/~slacoste/teaching/ift6269/A19/notes/lecture5.pdf
I always learned: "risk is the expected loss". In these lecture notes I see two ...
- 133
1
vote
0
answers
65
views
Cross-Validation estimate for the risk is almost unbiased
Let
$Z_N$ : set with N elements; full training set
$Z^l_{N/L}$ : set with N/L elements; l-th hold-out set
$Z_{N(1-1/L)}$ : set with N-N/L elements; e.g. 4/5 of data
$Z_N \setminus Z^l_{N/L}$ : ...
- 51
5
votes
1
answer
283
views
Mathematics of Random Forest Classifier
I was going through the this 2001 paper on Random Forest Classifier (RFC). I understood most of the concepts but there are some probability equations that I am finding hard to understand.
Definitions
...
- 55
3
votes
2
answers
433
views
What is accepted practice for avoiding optimistic bias when selecting a model family after hyperparameter tuning?
This is an extension of a previous question:
How to avoid overfitting bias when both hyperparameter tuning and model selecting?
...which provided some options for the question at hand, but now I would ...
- 308
10
votes
2
answers
5k
views
How to avoid overfitting bias when both hyperparameter tuning and model selecting?
Say I have 4 or more algorithm types (logistic, random forest, neural net, svm, etc) each of which I want to try out on my dataset, and each of which I need to tune hyperparameters on.
I would ...
- 308
9
votes
1
answer
724
views
AIC model averaging when models are correlated
AIC model-averaging: In "standard" AIC model averaging we average models with weights proportional to
$$w_i \propto \exp( -0.5 \times \Delta \text{AIC}_i ),$$
where $\Delta \text{AIC}_i$ is ...
- 38k
0
votes
2
answers
521
views
Is it Valid to Grid Search Cross Validation for Model Hyperparameter Selection then a separate Cross Validation for Generalisation Error?
The question has to do with Model Selection and Evaluation
I'm trying to wrap my head around the scale of how different nested cross validation would be from the following:
Let's say I am attempting ...
1
vote
1
answer
99
views
Why do neural networks outperform SVMs on image recognition if SVMs have the less generalization error?
Why do neural networks outperform SVMs if SVMs have the less generalization error according to Vapnik?
Is generalization error only useful in data scarce environments?
Is it because neural networks ...
user294054
5
votes
1
answer
148
views
what does it mean that there is leakage of information when one uses a test set?
I have read about the term "leakage of information" that occurs when one tries to estimate the generalization error by using a test set in Machine Learning models. However, I was not able to ...
- 641
1
vote
0
answers
42
views
Model performance metric on the test sample
Since usually k-fold cross validation is carried out on the training sample I understand how the mean and the standard deviation of a metric are computed for the training sample but how is the mean ...
- 11
1
vote
0
answers
183
views
Classification with noisy labels, noise is structured and not random
I am building a classification model with mislabeled training data on the order of ~70% of the training data is labeled correctly and ~30% is labeled incorrectly. Knowing this, how can I quantify the ...
- 1,648