Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty
Research output: Book/Report › Report › Research
Standard
Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty. / Merhout, Friedolin; Breznau, Nate.
2021. 19 p.Research output: Book/Report › Report › Research
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - RPRT
T1 - Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty
AU - Merhout, Friedolin
AU - Breznau, Nate
PY - 2021
Y1 - 2021
N2 - This study explores how analytical choices of researchers affect the reliability of scientificfindings. Current lack-of-reliability discussions focus on systematic biases. We broaden the lensto include idiosyncratic decisions in data analysis that lead researchers to diverging results andconclusions. We coordinated and observed decisions among 73 research-teams as theyindependently tested the same hypothesis using the same data. Results show that in this typicalsecondary data research situation, the universe of pathways from data to results is so vast that eachanalysis was unique in some way. Teams reported divergent findings with contradictorysubstantive implications that could not be explained by differences in researchers’ expertise, priorbeliefs, and expectations. This calls for greater humility and clarity in presentation of scientificfindings. Idiosyncratic variation may also be a cause for why many hypotheses remain highlycontested, particularly in large-scale social and behavioral research
AB - This study explores how analytical choices of researchers affect the reliability of scientificfindings. Current lack-of-reliability discussions focus on systematic biases. We broaden the lensto include idiosyncratic decisions in data analysis that lead researchers to diverging results andconclusions. We coordinated and observed decisions among 73 research-teams as theyindependently tested the same hypothesis using the same data. Results show that in this typicalsecondary data research situation, the universe of pathways from data to results is so vast that eachanalysis was unique in some way. Teams reported divergent findings with contradictorysubstantive implications that could not be explained by differences in researchers’ expertise, priorbeliefs, and expectations. This calls for greater humility and clarity in presentation of scientificfindings. Idiosyncratic variation may also be a cause for why many hypotheses remain highlycontested, particularly in large-scale social and behavioral research
M3 - Report
BT - Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty
ER -
ID: 292057602