video1.0<iframe src="https://www.loom.com/embed/6a1f47f8ff0b40568744e3bd66685143" frameborder="0" width="1920" height="1440" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>14401920Loomhttps://www.loom.com14401920https://cdn.loom.com/sessions/thumbnails/6a1f47f8ff0b40568744e3bd66685143-1886ca066e38b4d1.gif286.555This Loom presents a statistical significance layer for the BrainTrust platform to determine whether prompt changes that raise eval scores reflect real improvements or noise. The speaker shows that dashboards may display small gains like 62% to 64%, which can be misleading, especially in paired factual Q&A tests where a “clear win” improved by about 16% across 20 cases. They then demonstrate a deceptive case where scores rose by roughly 2%, but a paired bootstrap significance check yields a confidence interval indicating the result is not significant. The script pulls per-question scores via the BrainTrust SDK and computes confidence bounds from paired bootstrapping, concluding that raw score deltas can mislead shipping decisions without this gate.