Analytic ﬂexibility is known to inﬂuence the results of statistical tests, e.g. eﬀect sizes and p-values. Yet, the degree to which ﬂexibility in data-processing decisions inﬂuences the reliability of our measures is unknown. In this paper I attempt to address this question using a series of reliability multiverse analyses. The methods section incorporates a brief tutorial for readers interested in implementing multiverse analyses reported in this manuscript; all functions are contained in the R package splithalf. I report six multiverse analyses of data-processing speciﬁcations, including accuracy and response time cutoﬀs. I used data from a Stroop task and Flanker task at two time points. This allowed for an internal consistency reliability multiverse at time 1 and 2, and a test-retest reliability multiverse between time 1 and 2. Largely arbitrary decisions in data-processing led to diﬀerences between the highest and lowest reliability estimate of at least 0.2. Importantly, there was no consistent pattern in the data-processing speciﬁcations that led to greater reliability, across time as well as tasks. Together, data-processing decisions are highly inﬂuential, and largely unpredictable, on measure reliability. I discuss actions researchers could take to mitigate some of the inﬂuence of reliability heterogeneity, including adopting hierarchical modelling approaches. Yet, there are no approaches that can completely save us from measurement error. Measurement matters and I call on readers to help us move from what could be a measurement crisis towards a measurement revolution.