In deep and machine learning, we often do not have access to the underlying data distribution \(p_{\text {data}}\) that we believe generated the data of interest. However, we are often interested in maximizing the likelihood, or equivalently the log-likelihood, of the data under a parametric model, which involves an expectation such as \( \mathbb {E}_{p_{\text {data}}}\left [\log p_{\theta }(x)\right ]\).
Recall that, for a function \(f(x)\), the expectation is defined as
when \(p\) is a probability mass function, or as
when \(p\) is a probability density function. In many cases, these quantities cannot be computed directly or do not admit a closed-form solution.
Monte Carlo is a stochastic method for approximating such expectations using samples. Given a dataset of \(N\) samples \(x_i \sim p_{\text {data}}\), we can approximate
which converges to \(\mathbb {E}_{p_{\text {data}}}[f(x)]\) as \(N \to \infty \) by the law of large numbers.
A single-sample Monte Carlo estimator of \(\mathbb {E}_{p_{\text {data}}}[f(x)]\) is \(f(x_i), x_i \sim p_{\text {data}}\); while unbiased, it has high variance.