Kernel-based External Validity of Experimental Studies

Key points

• The external validity of a study is the extent to which its findings generalize to unseen experimental conditions, such as datasets.1
• Additionally, we establish an asymptotic power-law relation between the size of a study and the quantiles of the MMD, which we use to estimate the desired size of an externally valid study from a small-scale preliminary study.1
• We deploy our framework to rankings with ties and discuss kernels suited to different research questions.1
• We release our code as part of the genexpy Python module.1

Sources (1)

• [1]Kernel-based External Validity of Experimental Studies
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:08 PM

  We formalize the results of a study together with its research question, and define external validity as the probability that two independent and same-sized replications of the study produce results whose maximum mean discrepancy (MMD) is below a given tolerance.

  The external validity of a study is the extent to which its findings generalize to unseen experimental conditions, such as datasets.

Extractive summary: sentences quoted from the sources.