Visualising two approaches to explore reliability-power relationships

Abstract

The relationship between measurement reliability and statistical power is a complex one. Where reliability is defined by classical test theory as the proportion of ‘true’ variance to total variance (the sum of true score and error variance), power is only functionally related to total variance. Therefore, to explore direct relationships between reliability and power, one must hold either true-score variance or error variance constant while varying the other. Here, visualisations are used to illustrate the reliability-power relationship under conditions of fixed true-score variance and fixed error variance. From these visualisations, conceptual distinctions between fixing true-score or error variance can be raised. Namely, when true-score variance is fixed, low reliability (and low power) suggests a true effect may be hidden by error. Whereas, when error variance is fixed, high reliability (and low power) may simply suggest a very small effect. I raise several observations I hope will be useful in considering the utility of measurement reliability and it’s relationship to effect sizes and statistical power.

Type
Sam Parsons
Sam Parsons
Postdoctoral Research Associate

Researcher, Open Science Enthusiast, Lovable Bawbag

Related