Last updated Aug 2026 · Statistics
Why the log-likelihood of a Bayesian network decomposes across its CPDs when the graph structure is fixed and the data are fully observed, and what that decomposition buys us.
Suppose we fix the graph structure $G$ of a Bayesian network in advance, for instance the chain $I \rightarrow T \rightarrow A$ used in the earlier notes, and we want to learn the CPD parameters from data rather than specify them by hand. We observe a dataset $\mathcal{D} = \lbrace x^{(1)}, \dotsc, x^{(m)} \rbrace$ of $m$ fully observed, independent samples from the true distribution. The likelihood of the dataset under parameters $\theta$ is
\[L(\theta; \mathcal{D}) = \prod_{j=1}^{m} p_\theta\left(x^{(j)}\right),\]and maximum likelihood estimation chooses $\hat\theta = \arg\max_\theta L(\theta; \mathcal{D})$. As with most likelihood problems, it is more convenient to work with the log-likelihood, which turns the product over samples into a sum without changing the maximizer, since $\log$ is monotonic.
This is the key structural fact for learning Bayesian networks. Because the network factorizes as $p_\theta(x) = \prod_i p_{\theta_i}(x_i \mid x_{A_i})$, where $\theta_i$ denotes the parameters of node $i$’s CPD, the log-likelihood of a single sample is itself a sum over nodes:
\[\log p_\theta\left(x^{(j)}\right) = \sum_{i=1}^{n} \log p_{\theta_i}\left(x_i^{(j)} \mid x_{A_i}^{(j)}\right).\]Summing over the whole dataset and swapping the order of the two sums,
\[\ell(\theta; \mathcal{D}) = \sum_{j=1}^{m} \sum_{i=1}^{n} \log p_{\theta_i}\left(x_i^{(j)} \mid x_{A_i}^{(j)}\right) = \sum_{i=1}^{n} \left[\sum_{j=1}^{m} \log p_{\theta_i}\left(x_i^{(j)} \mid x_{A_i}^{(j)}\right)\right].\]The full log-likelihood is a sum of $n$ terms, one per node, and each term depends only on that node’s own parameters $\theta_i$. As long as the parameter sets of different CPDs do not overlap, which is the usual assumption, maximizing the sum is the same as maximizing each term separately. Learning a Bayesian network with fixed structure and full observability reduces to $n$ independent, small estimation problems, one per CPD, rather than one large joint optimization.
Worked example. For a categorical CPD $p(x_i \mid x_{A_i} = a)$, write $\theta_{i, v \mid a}$ for the probability that $x_i = v$ given parent assignment $a$. Maximizing the per-node log-likelihood subject to $\sum_v \theta_{i, v \mid a} = 1$ for each $a$ (via a Lagrange multiplier, or directly by noting the log-likelihood is a sum of independent multinomial log-likelihoods, one per parent assignment) gives the familiar count-based estimator
\[\hat\theta_{i, v \mid a} = \frac{\text{count}(x_i = v,\ x_{A_i} = a)}{\text{count}(x_{A_i} = a)}.\]In words: the maximum likelihood CPD entry is just the empirical conditional frequency in the data. Nothing more sophisticated is needed once the structure is fixed and the data are complete, which is precisely because of the decomposition above.
The empirical-frequency estimator is unbiased but can be unreliable when a particular parent assignment $a$ is rare in the data: with few or zero samples for $x_{A_i} = a$, the estimate $\hat\theta_{i, v \mid a}$ is high variance, or the count ratio is $0/0$ and undefined outright. This is what motivates treating $\theta$ itself as a random variable with a prior, typically a Dirichlet prior over each CPD row, and using the resulting Bayesian estimate (equivalent to adding pseudo-counts, sometimes called Laplace smoothing) instead of the raw maximum likelihood estimate. That is a natural next note.