Step 1 – Marginal (evidence) for a candidate design

Integrate the parameter out:

$$p(y\mid\xi)=\int p(y\mid\theta,\xi)\,p(\theta)\,d\theta$$

This is the predictive distribution of the not-yet-observed outcome. It’s the only place the prior enters, and it’s what makes EIG computable before running the experiment.

Step 2 – Posterior you would get, for a hypothetical $y$

$$p(\theta\mid y,\xi)=\frac{p(y\mid\theta,\xi)\,p(\theta)}{p(y\mid\xi)}$$

Step 3 – Information gain for that one outcome

$$\mathrm{IG}(\xi,y)=D_{\mathrm{KL}}\!\big(p(\theta\mid y,\xi)\,\|\,p(\theta)\big)=\int p(\theta\mid y,\xi)\log\frac{p(\theta\mid y,\xi)}{p(\theta)}d\theta$$

Careful point worth having in your back pocket: the “entropy reduction” form $H[p(\theta)]-H[p(\theta\mid y,\xi)]$ is not equal to this KL for a single $y$ – a surprising outcome can even increase posterior entropy. They coincide only after averaging over $y$ (Step 4).

Step 4 – Average over outcomes you haven’t seen yet

$$\;\mathrm{EIG}(\xi)=\mathbb{E}_{p(y\mid\xi)}\big[D_{\mathrm{KL}}(p(\theta\mid y,\xi)\|p(\theta))\big]\;$$

Substituting Bayes’ rule and swapping the order of integration gives the three equivalent forms you can quote on the slide:

$$\mathrm{EIG}(\xi)=\underbrace{\mathbb{E}_{p(\theta)p(y\mid\theta,\xi)}\!\left[\log\frac{p(y\mid\theta,\xi)}{p(y\mid\xi)}\right]}_{\text{likelihood form -- the one you actually compute}}=\underbrace{H[p(y\mid\xi)]-\mathbb{E}_{p(\theta)}H[p(y\mid\theta,\xi)]}_{\text{predictive entropy} - \text{noise entropy}}=\underbrace{I(\theta;y\mid\xi)}_{\text{mutual information}}$$

So EIG is the mutual information between parameters and outcome under the design. The middle form is the useful intuition: pick the experiment whose outcome you are most uncertain about overall, minus the part of that uncertainty that is just measurement noise. Ambiguity is good; noise is not.

The design step is then $\xi^\star=\arg\max_\xi \mathrm{EIG}(\xi)$.

Step 5 – Actually evaluating it

$p(y\mid\xi)$ is an intractable integral, so this is a nested expectation. Standard estimator (nested Monte Carlo):

$$\hat{\mathrm{EIG}}=\frac{1}{N}\sum_{n=1}^{N}\left[\log p(y_n\mid\theta_n,\xi)-\log\frac{1}{M}\sum_{m=1}^{M}p(y_n\mid\theta_{n,m},\xi)\right]$$

with $\theta_n\sim p(\theta)$, $y_n\sim p(y\mid\theta_n,\xi)$, and inner contrastive draws $\theta_{n,m}\sim p(\theta)$.

• Bias is $O(1/M)$ and positive (Jensen on the inner log) – it flatters your design. Optimal budget scaling is $M\propto\sqrt{T}$, $N\propto\sqrt{T}$ for total cost $T$.
• Cheaper variants: PCE/InfoNCE-style bounds, Barber–Agakov variational lower bound, VNMC upper bound. Sandwiching with a lower + upper bound is the honest way to report it.

The formula here works like this,

• For the first term, sample $N$ of $\theta$ values and for each of the $\theta$ values, sample $y_n$ which is composed of the computed exact value from the surrogate or physics mocel and the noise (e.g., Poisson noise). For the sampled $y_n$, calculate its probability $p(y_n\mid\theta_n,\xi)$.

• For the second term, we stay with the sampled $y_n$ in the previous step and we sample $M$ of $\theta$ values and we compute the probability to obtain $y_n$ for each of the sampled $\theta$ values, giving us $p(y_n\mid\theta_{n,m},\xi)$.