• Question: How common is it for researchers to replicate their own work?

    Asked by Abena to Sara, Richard, Paula, Mike, Courtney, Carolina, Brian, Alex on 12 Mar 2018.
    • Photo: Richard Churches

      Richard Churches answered on 12 Mar 2018:

      Replication in education research is a rare thing… but essential to scientific method. In the 19 teacher trials we have several teachers who are doing parallel replications. One teacher has trialled spaced learning in KS1 and KS2 lessons in parallel to look at the effects of the approach on different age groups and with different lesson content.

    • Photo: Paula Clarke

      Paula Clarke answered on 12 Mar 2018:

      I think it is very common, especially with smaller scale experimental work. It is more challenging with RCTs as they are typically expensive to run and it can take time to see the impact of intervention. However, I guess I am thinking about it from the perspective of psychological research – I agree replication seems to be less common in education research.

    • Photo: Brian Butterworth

      Brian Butterworth answered on 12 Mar 2018:

      Excellent question. In the best cases, we try to replicate within the experiment, rather than in a separate experiment. For example, if we have a large sample, we can divide it in two, and see if the same effect applies in both halves. In genetic experiments, it is standard procedure nowadays to try to replicate in a different sample of people. Sometimes, we will try a slightly different procedure to see if it comes up with a concordant result: this a way of testing whether our explanation is a valid construct.

    • Photo: Alex Hodgkiss

      Alex Hodgkiss answered on 13 Mar 2018:

      Just following up with what Brian said on different procedures with an example (although this is from previous associational research rather than an RCT). In an initial study, I found a link between spatial cognition and science achievement in the 7-11 age range; this was based on broad curriculum-based science measures. The recent follow up study looked at the relationship in a more fine-grained manner, based on children’s participation in a science lesson on a specific topic, but with a similar spatial tasks. The findings of the second study mapped onto the first, e.g, the same spatial task was the best predictor, but went further (e.g, took into account a greater range of possible confounds), and was more valid to the classroom by doing an assessment following learning. So not a direct replication but replicated many of the results.

    • Photo: Carolina Kuepper-Tetzel

      Carolina Kuepper-Tetzel answered on 13 Mar 2018:

      Wonderful question that is impossible to answer for two reasons:

      1) For the longest time, if you replicated a finding, it would have been impossible to publish it because it was just a replication. Things in research are changing in that respect, but the change is slow.
      2) Not replicating a finding – in most cases probably revealing a null finding – are extremely hard (read: impossible) to publish because again the mindset in research for a long time now was that non-significant effects are not worth reporting. So, non-significant findings went into the file drawer.

      In both cases, successful and unsuccessful replication attempts could usually not be published and doing replication studies were not rewarded by the scientific community. As a consequence many researchers would decide to stay away from replications altogether. Taken together, no one knows what the rate of replication actually is. Personally, I think, this is a shame and many other researchers think so too. So currently there is a great push towards a more transparent approach to replicability and research, in general. I hope that with this new movement things will get clearer in the future.