2
$\begingroup$

I'm working through Dive Into Deep Learning right now and am struggling with the following question:

  1. We can explore the connection between exponential families and the softmax in some more depth.
    1. Compute the second derivative of the cross-entropy loss for the softmax.
    2. Compute the variance of the distribution given by $softmax(\bf{o})$ and show that it matches the second derivative computed above.

For part 1, I've already calculated the 2nd derivative as $softmax(\bf{o})_j(1-softmax(\bf{o})_j)$, where $\bf{o}$ is a vector given by $\bf{Wx+b}$, where $\bf{x}$ is a vector of inputs, and $\bf{W,b}$ are weights and biases.

For part 2, I am unsure where to begin. I know $Var[X] = E[X^2] - E[X]^2$, and $E[X] = \Sigma xf(x)$ or $\int xf(x)dx$, where $f(x)$ is the pdf. But in this case, I'm not really sure what $x$ would be other than the output of the softmax function, and I'm not sure how to determine the pdf of that.

Improve this question
$\endgroup$
2
  • $\begingroup$ Welcome to the site! "Compute the variance of the distribution given by and" seems to be missing a word or symbol. $\endgroup$ Commented Jul 28, 2022 at 20:02
  • $\begingroup$ Hi John, yes thank you. The LaTeX didn't copy over. I have updated the original post $\endgroup$ Commented Jul 29, 2022 at 1:30

1 Answer 1

Reset to default
1
$\begingroup$

I think the distribution given by softmax(o) is a Bernoulli distribution with probabilities $p = softmax(o)$, so the variance is: $$Var[X] = E[X^2] - E[X]^2 = \text{softmax}(o)(1 - \text{softmax}(o))$$ details are here

Improve this answer
$\endgroup$
1
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$ Commented Aug 17, 2023 at 3:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.