ModKathryn: Hi! Welcome to tonight’s live chat on Evidence in the classroom. We’re due to start at 8pm, but it would be nice to introduce ourselves!

Abena: I’m most recently an international school teacher of English and Digital Literacy, (hopefully) starting a Psychology conversion course in September. Also currently studying Research Methods with Coursera.

ModKathryn: My name is Kathryn, I’m a moderator for the Learning Zone and a PhD student in cognitive development

Richard: Hi, I’m Richard. I have been teaching teachers to do randomised controlled trials for the last three years and am now working on a Wellcome Trust project in which teachers who have previously completed an RCT have designed trials in collaboration with neuroscientists.

Paula: I’m Paula, and I am currently working on developing a previous RCT as I hope to replicate and extend the findings we obtained. The work I hope to develop is on developing reading comprehension skills through oral language training

Abena: @Paula – how common is it for researchers to replicate their own work?

Richard: Replication in education research is a rare thing… but essential to scientific method. In the 19 teacher trials we have several teachers who are doing parallel replications. One teacher has trialled spaced learning in KS1 and KS2 lessons in parallel to look at the effects of the approach on different age groups and with different lesson content.

Paula: @abena – I think it is very common, especially with smaller scale experimental work. It is more challenging with RCTs as they are typically expensive to run and it can takes time to see the impact of intervention

Paula: @all I guess I am thinking about it from the perspective of psychological research, I agree replication seems to be less common in education research

Abena: Why are RCTs so expensive?

Richard: Actually, the large scale trials in England are only expensive because of the way they are organised. One of our teacher-led RCTs has 900 participants and has cost nothing more than teacher time

Paula: RCTs don’t have to be expensive but when run by a research team the costs mount up

Richard: Also, the large trials in education are trying to standardise their protocols over six months with large samples. Teacher-led RCTs often model a laboratory approach with many being single lesson studies

Paula: When we conduct trials we will typically employ research assistants, postdoctoral researchers, pay for buy out of teacher time and teaching assistant time, assessment costs, materials costs etc.

Abena: Do researchers across the globe share and pay attention to each other’s studies? Or do they tend to be more nationally-focused (or even US-UK focused in this case)?

Paula: @abena the number of researchers who work in the field of reading comprehension intervention in the UK is pretty small, it is really important to connect with research internationally

Abena: @Paula – do these connections tend to stay in the ‘western’ world or truly internationally?

Courtney: @abena I think in theory researchers are attending to other’s work, but this may vary by topic. For instance, there may be many researchers in a particular geographic region or country with a similar focus

Paula: Connections are maintained through publication in international journals, conferences, online and through teaching – I have students on the MA programmes at Leeds from around the globe and PhD and EdD students working in international contexts

ModKathryn: @all What are some of the key challenges in translating evidence to practise in the classroom? And how do you think we can, or we are, addressing them in education research?

Richard: Firstly, laboratories are not classrooms, just as the biological experiment is not clinical practice. Secondly, wide replication to control for individual pupil differences as well as school context will be necessary. Finally, and most importantly, writers have pointed to the ‘democratic deficit’ that exists in education research and its potential impact on attempts to establish ‘what works’ (Biesta, 2007). In medicine and healthcare, it is serving clinicians who most frequently publish studies about clinical practice. In education, few practitioner studies reach journals or get disseminated. Further, those researchers who do study or design pedagogy often no longer practise as teachers.

Courtney: I think one challenge is moving from a lab setting into a classroom – to think about how a hypothesis may need to be tested under different circumstances (and revised and tested again)

Alex: Teachers are extremely busy – and they seem to be getting more overworked all the time, and see new fads and changes to curriculum etc all the time – so any additional techniques/methods etc can be a challenge, amongst everything else they need to do

Richard: Another key challenge in relation to translating neuroscience evidence is this. . .on a theoretical level, education and neuroscience can be considered as fundamentally different in their overall objectives and the manner in which the objectives are pursued (the ‘goal problem’). Neuroscience is a natural science that investigates the workings of the brain, the functional architecture of the mind and the way that the brain and mind map together. In contrast, education aims to develop particular pedagogies and therefore, arguably, has more in common with the way in which architecture uses physics.

Abena: @Richard – do you think EdD programs are an answer to the practitioner / researcher issue?

Richard: Few EdD programmes in education include experimental research methods, yet. Education research has tended to be dominated by qualitative methods

Paula: Our EdD programme at Leeds gives the option of doing advanced experimental and statistical methods

Abena: @Richard – you see the qualitative slant as a problem?

Richard: The qualitative slant. . well,. observation alone can’t tell you if something is working or not. Without a control condition you have no idea if the effect you have observed would have happened anyway. After 100 year of education qualitative research we still can’t answer some basic questions about which pedagogy is best with which group in which context

Abena: @Richard – Yes, I see what you mean. I hadn’t realised qual. research didn’t include control groups. Just beginning in my studies in this area

Abena: I’m baffled as to how the idea of ‘evidence-based practice’ is so neglected in ITT programs and PD

ModKathryn: What do ITT and PD stand for?

Abena: Initial teacher training and professional development. I’m just surprised at the distance between research and teacher knowledge of it.

Richard: You might find this article interesting. https://impact.chartered.college/article/dommett-bridging-evidence-classroom-teacher-rcts/

Alex: @abena Absolutely agree – this is an issue for ITT as you say. But also for school-based CPD

Paula: In my work I am moving more and more to phases of co-production, working in partnership with teaching assistants to develop materials and training @Richard I think your work sounds very interesting

Alex: The point I was making was more that, teacher’s time or lack of is a challenge for translating research into the classroom – particularly when they have a way that seems to ‘work

Richard: You can find some examples of teacher-led RCTs here

Abena: @Alex – yes I agree about the time demands for sure. Since learning about orgs such as this and EEF, I want everyone to know so we stop pouring energy into black holes.

Courtney: @Alex @abena There have been discussions about creating space for research-practice translators in schools, but how to do this has been a challenge.

Richard: I think the large scale EEF trials are helpful. However, its going to take a long time to answer all of the questions we need to… I am also looking at developing RCTs which involve the collection of quantitative and qualitative data – so key outcome measures which may be psychometric and quantifiable and outcomes which can captured using reflective diaries, interviews etc. However, if only 10% of teachers in England led one robust RCT onece in their career we could have 40,000 studies.

Abena: @Courtney – presumably they would be teachers? (The translators)

Courtney: They could be, or could be former educators or researchers who want to do the translational work. For sure they would need training through schools of education or in-school training.

Paula: My most recent trial was funded by EEF. I think what is important is that there is an opportunity for re-trialling and developing cycles of investigation. I am particularly interested in the potential application of design based methods.

Richard: You can run a qualitative RCT. It is sometimes done in medicine to explore the qualitative impact of a treatment alongside the patient outcomes

Abena: @richard what about the research schools – is that not what they are aiming for? Or are those studies too small-scale?

Richard: That is the main issue with large-scale expensive trials. When we conduct such research we only end up with a probability that the result may have occurred by chances. Without replication you do not know if the finding is the 1 in 20 anomaly

Courtney: @Paula I think the mixed-methods approach is really interesting and valuable – provides such a rich data set.

Abena: Another obstacle (I guess) is the (seemingly to me) dominant culture of dismissing further studies (e.g. teachers gaining masters) as for self-interest, rather than the furtherance of or understanding of pedagogy. Just last week on TES there was a thread shouting down a teacher who said s/he interested in doing a PhD in education. A bit like the kid in the class who puts his hand up and then gets teased (or bullied) for it later

Richard: That is sad

Alex: do you think the majority of people who do masters are people who want to or aspire to management positions?

Courtney: What was the reason for that? (also, I may have missed what TES is)

Abena: @Alex – I honestly don’t know. I’ve met many US teachers who did it because of the competitiveness for good jobs, but it *seems* less common in the UK. TES = Times Educational Supplement. They run a forum for teachers

Alex: There was talk a while back about making teaching a masters level profession I think – masters in teaching and learning was it? Which would presumably include research elements, but I think the funding wasn’t there

Abena: @Alex – yes. Many PGCEs are now counted as the first part of this masters. Not sure what the funding status is now but I think at one point teachers could get funding to do it from their LEA

ModKathryn: I think Prof Paul Howard Jones is doing some work in collaboration with wellcome trust about implementing more information on brain development and the science of learning into teaching training – I heard him mention at a talk. Worth keeping an eye out for!

Alex: Oh yes, extra modules are being trialed for ITT

Abena: Where are these modules being trialled do you know?

Alex: Bath springs to mind

ModKathryn: Also Bristol – where Howard Jones is based.

Abena: @Richard – why do you think this difference exists (between healthcare and education)?

Richard: I think that there is a fundamental issue beyond this, related to the content of initial teaching training and masters degrees in education – which is not necessarily evdidence based. I think the difference is there because education is still an area at the mercy of fads and quackery

Abena: I agree! I have a masters and hadn’t been exposed to many of the ideas I’ve found recently through this (and related sites). Quite annoyed about it but glad I can move forward now

Paula: We run an MA Education and Professional Enquiry which is specifically for teachers, many of the students use this as a route to a PhD or EdD. The MA in Special Education Needs which I lead on also requires a minimum of 3 years teaching experience.

Richard: As the Journal Trends in Neuroscience and Education puts it – “Just as 200 years ago, medicine was little more than a mixture of bits of knowledge, fads and plain quackery without a basic grounding in a scientific understanding of the body, and just as in the middle of the nineteenth century, Hermann von Helmholtz, Ernst Wilhelm von Brücke, Emil Du Bois-Reymond and a few others got together and drew up a scheme for what medicine should be (i.e. applied natural science), we believe that this can be taken as a model for what should happen in the field of education. In many countries, education is merely the field of ideology, even though we know that how children learn is not a question of left or right political orientation.”

Richard: There are many places now where educational neuroscience is becoming a core part of academic activity

Richard: Another issue, is that we have no formal pipeline equivalent to the ‘bench to bedside’ concept in medicine and healthcare

Abena: @Richard – I don’t know what that is, but is anyone pushing for the creation of such an equivalent, that you know of?

Richard: Even if a new ‘treatment’ were discovered in education there is no guaranteed way of getting that information into teacher training (as the whole system is fragmented and devolved)…. it is even possible become a qualified teacher with no academic input at all

ModKathryn: Also behavioural research takes a lot longer to get to valid, replicable outcomes which can be rolled out in large-scale interventions

Paula: yes I agree @modkathryn

Richard: Yes that is true too

Richard: I am on the organsing group of CEBE (Coalition for Evidence Based Education) and this year we are focusing on the differences between medicine and education . . in relation to this ‘pipeline’ question

Courtney: @Richard What take-aways may come out of the CEBE meeting? Would the CEBE make recommendations, for example, or a framework to address the pipeline issue?

Richard: Yes that is the plan

Richard: ‘bench-to-bedside’ in healthcare expresses a linkage from basic lab research through applied research and evaluation of programmes to evidence-based treatments and training to deliver them. My understanding is that these functions were all carried out by different bodies in separate silos, as in education, but after a lot of effort they have been linked by the creation of the National Institute for Health Research. We are a long way from this in education

Abena: @Richard – any word of an equivalent being established in education, or is that not the gap the EEF is trying to plug? Or even the CEBE?

Richard: Here is a theory… people often feel very defensive of their PhD research method… and many of the last generation of education researchers were ethnographic sociologists

Richard: In all our teacher-led RCTs. . . the control condition is existing best practice… with the intervention a ‘believed’ improvement. As would be done in surgery. Therefore, in the control condition the teachers are doing no more harm than usual.

Abena: @Richard – don’t think-tanks bring that kind of expertise and knowledge to policy-makers?

Richard: It is a bit of a chicken and egg problem. education is so highly politicised precisely because we don’t have enough evidence. . . and we probably don’t have enough evidence because politicians have spent too long shifting from one position to another…. the answer is to do the hard work. . . the controlled research.

Paula: @Richard – same – we use a ‘business as usual’ control – we also have a waiting list element though so that those who did not receive intervention in the trial can have the opportunity to receive it after the trial is complete.

Richard: About 40% of our trials are within-participant (repeated measures) so that the children experience both control and intervention conditions . .. with lesson content as well as the order of conditions counterbalanced

Courtney: @Paula @Richard It seems that a delayed treatment (i.e., waiting list) helps with retention for classroom-based RCTs

Richard: Sometimes you can only do a between-participant design (independent measures) so a wait-group is a good solution to avoid high levels of attrition in the control group…this is what we did on Closing the gap

Richard: Just to clarify on what this means: Between-participant – pupils divided into two groups that each experience a different teaching approach (or condition).Within-participant – all pupils experience all approaches but in different orders (counterbalancing). It is usual to counterbalance the order in which things happen in this type of design, in order to balance out effects that might transfer from one condition to another (carryover or order effects). Matched-pair (or case-matched) – similar pupils are paired and each member of the pair randomly allocated.

Paula: Initial teacher training and postgraduate level training has been mentioned so far in this conversation – I am interested in cost effective ways to deliver continued professional development at scale. I would be grateful for any suggestions of organisations to link up with. I am also interested in online and blended learning approaches to CPD if anyone has any advice there?

Abena: @Paula – my last school had a few cohorts doing online CPD with Harvard’s programs for teachers

Paula: thanks – what kinds of topics do you think work well in these types of training packages?

Abena: @Paula – in all honesty, I think it’s the content that allows for easy implementation, with minimal additional resources. And anything that saves time rather than creates more work is – of course – most popular. I’ve heard many teachers saying that after their masters they never want to study again because of the impact on their time and especially families / partners. But where schools make space for this, it is more popular e.g. reducing timetables

Paula: @abena – thank you for these insights

Alex: abena: how do you think teachers perceive this type of education research?

Abena: @Alex – as you mentioned, it’s a time issue. And also knowing how to separate the wheat from the chaff. Those that experiment can be seen as using kids as guinea pigs; those who don’t can be seen as disinterested in professional development.

Abena: Communities such as this one are excellent for having access to experts. I’m wondering if any asynchronous forums for teachers to discuss research in the company of experts exist. Do you know of any?

ModKathryn: I know its not specific but twitter seems to be a good place to have these discussions between researchers and teachers

Richard: I was talking to Shane about how we might put some of the teacher protocols online here and give other teachers the chance to talk to the teacher who have done RCTs… and even replicate the studies, uploading data

Richard: My twitter is @teacherled_RCTs

Paula: My twitter use is growing and I intend to become more active on it – @DrPJClarke

Courtney: Yes, I hope to as well. @_CPollack

Alex: @hodgkiss_alex here

Abena: Thank you all for being so accessible!

Richard: Thanks for an interesting conversation. . so many of us are now in the same space. . . we definitely should connect more

ModKathryn: Thank you all for the discussion! The transcript will be posted online tomorrow with links attached. Remember if you have any other questions you can ask at any time using the “”Ask”” button on the website. See you in the next chat! 🙂