The entropy is a measure of a probability distribution’s average information content and is given
by:
\[ H(p) = \sum _x p(x)\log \left (\frac {1}{p(x)}\right ) = -\sum _x p(x)\log p(x) \]
The cross-entropy between two probability distributions \(p\) and \(q\) is defined as
\[ H(p,q) = -\sum _x p(x)\log q(x) \]
Given two probability distributions \(p(x)\) and \(q(x)\) over a discrete random variable \(X\), the Kullback–Leibler
divergence measure of the distance between them and is given by \(D(p\|q)\) is defined as follows:
\[ D_{KL}(p\|q) = \sum _{x\in \mathbb {X}} p(x)\log \frac {p(x)}{q(x)} \]
Observe that
\[ H(p,q) = H(p) + D_{\mathrm {KL}}(p\|q) \]
Also, maximizing the likelihood of the data is equivalent to minimizing the KL divergence
between the data distribution and the model distribution:
\[ \arg \min _{\theta } D_{\mathrm {KL}} \left ( \hat {p}_{\mathrm {data}}(x) \,\Vert \, p_{\theta }(x) \right ) = \arg \min _{\theta } \left \{ -\mathbb {E}_{\hat {p}_{\mathrm {data}}(x)} \left [ \log p_{\theta }(x) \right ] \right \} = \arg \min _{\theta } \left \{ -\frac {1}{N} \sum _{i=1}^{N} \log p_{\theta }(x_i) \right \} \]
Now, given the joint probability distributions \(p(x,y)\) and \(q(x,y)\) over two discrete random variables \(X\) and \(Y\), the
conditional KL divergence between the conditional probability distributions \(p(y\mid x)\) and \(q(y\mid x)\), averaged over \(p(x)\), is
given by
\[ \begin {aligned} D_{\mathrm {KL}}\left ( p(y\mid x)\|q(y\mid x) \mid p(x) \right ) &= \sum _{x\in \mathbb {X}} p(x) \sum _{y\in \mathbb {Y}} p(y\mid x) \log \frac {p(y\mid x)}{q(y\mid x)} \\ &= \sum _{x\in \mathbb {X}} p(x) D_{\mathrm {KL}}\left ( p(y\mid x)\|q(y\mid x) \right ) \\ &= \sum _{x\in \mathbb {X}} p(x) \left [ \sum _{y\in \mathbb {Y}} p(y\mid x)\log p(y\mid x) - \sum _{y\in \mathbb {Y}} p(y\mid x)\log q(y\mid x) \right ] \\ &= H\left ( p(y\mid x),q(y\mid x) \mid p(x) \right ) - H\left ( p(y\mid x) \mid p(x) \right ) \end {aligned} \]