Disagreement over how to interpret the results of a line of research often boils down to whether or not there is scope for researcher decisions (or bias) to influence the database. For example, the published database can be distorted if a researcher chooses to stop collecting data at an auspicious point, decides not to publish results that don’t come out as hoped, or analyses the data in various ways then only reports the analyses that give the desired outcome. One way to minimize the opportunities for researcher bias to operate is to deposit a statement of the planned number of trials, hypotheses and analyses prior to the collection of any data. This is known as study registration.
In 2012, Jim Kennedy and I launched the KPU Registry for Parapsychological Experiments. This is a simple registry that makes the researcher’s plans publicly available. It offers researchers, reviewers, editors and others the ability to identify studies that were planned for a line of research and to verify that a study report matches the pre-registered information.
In 2015, we published in Frontiers in Psychology recommendations for improving study registration practice based on the lessons we had learned from running the KPU Registry. These included the importance of distinguishing exploratory from confirmatory analyses, and the need for review of submissions to the registry to ensure the registered information unambiguously meets the registry requirements. Interestingly, our experience is that, despite us providing a template and guidelines for registration, several go-rounds of review and revision are typically needed to iron out ambiguities in registered information. This fact hints at the existence of loopholes in registries that do not review submissions, undermining the basic purpose of the registry.
In the last few years, psychologists have increasingly become aware of the dangers of researcher bias, and numerous steps have been taken to address these issues. To summarise and evaluate the main developments in this rapidly changing arena, Jim Kennedy and I have added two short comments to our Frontiers paper. In brief, we feel that study registration in psychology has made some noteworthy progress but is still sub-optimal. You can read our original Frontiers paper, and our comments evaluating the current situation, here.