2
$\begingroup$

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy

enter image description here

Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy between $p$ and $q$. But why is it called the cross-entropy of the distribution $q$ relative to a distribution $p$, not $p$ relative to $q$?

Also from this wikipedia page, cross entropy $H(p,q)$ can be written as

$$H(p,q)=H(p)+D_{KL}(p||q)$$

where $D_{KL}(p||q)$ is the KL divergence of $p$ from $q$, or the relative entropy of $p$ with respect to $q$. Now it is $p$ with respect to $q$？

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Wikipedia calls it $KL(P||Q)$ "the relative entropy from $Q$ to $P$", so this also could be interpreted as $P$ being the "reference distribution".

The cross-entropy can be understood as the average Huffman encoding length when the distribution is thought to be $q$ but is actually $p$, i.e. the "real thing" is $p$. So, it makes kind of sense to consider $p$ as the distribution that is the "baseline".

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you. But why is $D_{KL}(p||q)$ called the relative entropy of $p$ with respect to $q$ in both the cross entropy and KL divergence wiki page? $\endgroup$ Commented Jul 30, 2022 at 9:05
  • $\begingroup$ On Wikipedia I could only find the expression "the relative entropy from $Q$ to $P$". But maybe sometimes someone switches it, maybe unintentionally, maybe intentionally. Anyway, what really matters is that you know what the actual meaning is. $\endgroup$ Commented Jul 30, 2022 at 9:22
  • $\begingroup$ OK, thank you. If you search "with respect to" in that page you will see it at the end of the Definition section, which is written as "Another common way to refer to $D_{KL}(P||Q)$ is as the relative entropy of $P$ with respect to $Q$". $\endgroup$ Commented Jul 30, 2022 at 9:55
  • $\begingroup$ OK, I see. But this very paragraph seems to answer your question quite comprehensively, right? $\endgroup$ Commented Jul 30, 2022 at 10:02
  • $\begingroup$ Yes, thanks. I was just wondering how to correctly write it in my thesis. $\endgroup$ Commented Jul 30, 2022 at 10:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.