0
$\begingroup$

Suppose that we have a general loss function that depends on some parameters $w$ (e.g. neural network weights): $$L_w =\frac{1}{N} \sum_i \ell(\hat{y}_i, y_i)$$

Is it beneficial to standardize the target in addition to features?

That is, should we prefer to optimize $L_w'$: $$L_w' =\frac{1}{N} \sum_i \ell\left(\hat{y}_i, \frac{y_i-\bar{y}}{\sigma} \right)$$

over $L_w$?

Related questions

In the accepted answer of this question, it is stated that:

Normalizing the output will not affect shape of $f$, so it's generally not necessary.

where $\hat{y} = f_w (x)$. However, during training we optimize the loss function and as such, the shape of $f$ is irrelevant.

Improve this question
$\endgroup$
4
  • $\begingroup$ This will obviously depend on the loss function. Accordingly, this question makes little sense as stated. What specific loss function(s) do you care about? $\endgroup$ Commented Aug 4, 2024 at 19:23
  • $\begingroup$ @g g I am more interested in the effect of standardizing for gradient descent. I will edit the question appropriately. $\endgroup$ Commented Aug 4, 2024 at 22:42
  • 1
    $\begingroup$ Bear in mind your general loss function $\ell$ can absorb your intended output normalizing. And for consistency purpose if you normalize the ground truth output, then you also need to normalize the model predicted output in the same loss function. $\endgroup$ Commented Aug 12, 2024 at 2:31
  • $\begingroup$ @cinch We usually normalize just the ground truth values as a preprocessing step and don't modify model predictions (during training). So, should we opt for this preprocessing step or not? This is what I am asking. For example, when doing regression with gradient descent and MSE (mean squarred error) should we trained with the normalized ground truth values? $\endgroup$ Commented May 22 at 19:35

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.