video1.0<iframe src="https://www.loom.com/embed/749aea6dca2f4897b8a01b2b5852531e" frameborder="0" width="1110" height="832" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>8321110Loomhttps://www.loom.com8321110https://cdn.loom.com/sessions/thumbnails/749aea6dca2f4897b8a01b2b5852531e-cca9063883575a40.gif186.57This Loom explains how variant predictor benchmark results can be badly biased by non-random selection of “hotspot” and clinically seen variants and how inverse propensity weighting (IPW) fixes it. Using a pharmacogen set (T15 NUD 15) where predictors essentially disagree, the naive estimate has an error of nearly two tenths, with a benchmark batch bias around 0.29, which IPW reduces to about 0.01. Randomizing variant selection makes the bias vanish, and the same recovery happens for hotspot residues where labs focus on consensus important sites. The speaker also notes two pre-registered attempts to beat random selection, including one that became worse than random and a later point where random caught up by 768 variants.