“Great Buys” in Education: Maximizing learning in low and middle-income countries

In this episode, we are featuring ⁠Benjamin Piper⁠, the Director of Global Education at the Bill & Melinda Gates Foundation.

Ben sheds light on the groundbreaking work of the Global Education Evidence Advisory Panel (GEEAP). With an exhaustive analysis of over 13,000 academic studies, GEEAP identifies the most effective and economical educational interventions.

We delved into the intriguing concept of "Smart Buys" or "Great Buys" in education, dissecting approaches like structured pedagogy and teaching children according to their learning level rather than age or grade.

We confront a stark reality: two-thirds of children in low and middle-income countries fail to acquire comprehensive reading skills by age 10. This crisis extends beyond literacy; it reflects the profound limitations of our current education systems. The question arises: How do we tackle this challenge? It's essential to look beyond simply increasing school enrolment. We must ensure that every child, once in school, has equal opportunities to maximize their learning potential.

[What follows is a transcription of the podcast, modified for enhanced web readability.]

Paulyn Duman: Welcome to the SDG Learncast with me, Paulyn Duman. In every episode, I bring insightful conversations around the subject of sustainable development and learning, helping us all to achieve a sustainable future.

Are you an advocate for education in low and middle-income countries? Ever wondered how we can bridge the gap between the massive learning poverty and the potential of every child?

This SDG Learncast episode unpacks the recent evidence on the most effective educational interventions. Joining us in this episode is Mr. Benjamin Piper, the Director of Global Education at the Bill and Melinda Gates Foundation. In this podcast, we will unravel the secrets of what are called ‘Smart Buys’ in education.

Welcome to the SDG Learncast, Ben.

Benjamin Piper: Thank you.

Paulyn Duman: So let's begin with the Global Education Evidence Advisory Panel, which you are part of. Can you tell us about this panel and your role and why did you study 13, 000 studies on education?

Benjamin Piper: Fundamentally, this panel, the Global Education Evidence Advisory Panel, was convened in 2020. The basic thing it's trying to do is pull together leading education experts from around the world. We have panelists from all over. Actually, I'm a panelist myself.

It's cross-disciplinary. So it's not just people who have the same view of education. We have economists, pedagogical experts, and researchers of various types. It's co-hosted by FCDO, the British, UNICEF, USAID, and the World Bank.

The basic idea is that this is not new to education, but it's not new to development. Advisory panels are quite common in health; they're common in the climate area. This panel is trying to fill the gap in education to help policymakers better make use of the evidence that exists. These panelists are leading researchers and also practitioners who have contributed to an increasing evidence base in education, increasing in two different ways.

Number one, evidence is rigorous, so it's well-designed to be able to create a causal estimate, and it's rigorous with respect to cost-effectiveness, which is a newer but growing field in the education sector.

Let me boil it down. It's to provide succinct, hopefully, we'll see when you read it, usable and really policy-focused recommendations to help these countries make better decisions, particularly in low and middle-income parts of the world.

So the 13,000 studies were the starting point. We actually used 400 studies that hit the evidentiary bar. The most up-to-date, the most rigorous on cost-effective interventions that have been implemented and increasingly implemented at scale. The task of the panel was to do that analysis and then to present these complex and disparate research studies in a user-friendly way for policymakers.

Paulyn Duman: This paper is one of the most useful policy papers I've read. It's really well thought of, so congratulations on that. One of the things that really intrigued me was this concept of Smart Buys. Can you shed some light on this concept? Can you also share some really interesting real-world examples or case studies that embody the essence of Smart Buys?

Benjamin Piper: So the congratulations go to the research team and the other panelists who have really spent some time and effort, including the co-chairs. Albert G. Banerjee and Kwame Akyempong have done a great job leading, and Rachel Glenister has done a great job with intellectual leadership. So credit goes to them.

The basic idea of Smart Buys is that, look, I spend time with ministers and education leaders. We think our job as researchers and implementers is difficult. How much more difficult to be an education leader in the context of an economic downturn? And for a variety of reasons, including increasing debt burdens in the countries in this part of the world that I care about (I'm calling from Kenya, though I'm based in Ethiopia), where countries like these are experiencing limited fiscal space and falling overseas development assistance (ODA).

So fundamentally, governments using their own domestic resources are having to do more with less.

And they have to do more because outcomes are bad, and they have less because of fiscal constraints, so they have tight budgets. It's important that they get the most impact on children from the resources that they have. So the Smart Buys are a way to express solidarity and connection with these policymakers, with the panel putting itself into their shoes and not just looking at what works in education.

This is my field. I can tell you a lot about what I think is effective, but it's a very different thing to have a bar of high-quality rigorous evidence and also costs on what it looks like. Not just for researchers or pilots, but what does it cost for governments to deliver an intervention at scale and ensure sustainability.

So the report groups many different types of education policies and programs into categories if you've read the report from Great Buys to Bad Buys on the other end and has categories where there's not enough evidence, so more knowledge is needed.

And I think it's really critical for everyone to know that this idea of Smart Buys is targeted. It's Smart Buys for learning. So you could look at a lot of things, but I think the panel agrees that the most important thing is learning and learning outcomes. So it's how effective these various categories of interventions and very specific interventions are to improve learning most cost-effectively.

Paulyn Duman: Do you have some examples of Smart Buys?

Benjamin Piper: I wanted to highlight the two that I think have been upgraded. In this version of the report, there's a version of the report in 2023 that follows on from a pre-COVID 2020 report, and the things that have shifted in this report, in the Smart Buys, the Great Buys is actually what they're called. The Great Buys, the highest category, are two, and they're actually interesting because they have one thing in similarity, which is that they're focused on the classroom. They're focused on helping teachers to teach better.

The two new Great Buys are structured pedagogy, which I'll talk about in a second, and then teaching at the level of the child instead of their age or their grade. So let me tell you what structured pedagogy means.

This is about helping teachers teach better. The sector is showing that a lot of interventions call themselves structured pedagogy, but the panel defines this as a coherent, coordinated, and combined package that includes lesson plans for teachers, learning materials for kids, often at a one-to-one ratio, skill-based ongoing teacher training, and teacher mentoring or coaching that reinforces improved practice. But the key thing is that all four of those things—lesson plans, student materials, teacher training, and teacher support or coaching—all integrate. So you're not just training theoretically on random things in the area of literacy. You're training the teacher specifically on how to teach this often phonics-based, integrated literacy program more effectively. And just to emphasize that the most effective version of this for literacy often includes phonics-based approaches that provide step-by-step support to teachers.

Social pedagogy is not only in the literacy area. There are quite a few programs that are in numeracy as well.

But right now, we have strong evidence that this structured pedagogy approach can improve learning at a large scale and at low cost. And there's evidence for many countries for this: India, Kenya, Liberia, South Africa, amongst many others that have shown improvements in this sort of thing. Senegal, Ghana, Nigeria.

Just others that are coming to mind as we're talking.

A second one is called teaching the level of the child. One of the things that many countries struggle with is that there's a task for the curriculum, which is to teach the kids the particular things that the curriculum designers have decided are age-appropriate.

But often, given the weaknesses in systems and various other reasons, kids are not coming to that grade two, let's say, with the requisite skills that you assume. But the challenge for the teacher is, my job is to teach what this grade two curriculum says, and if my kids are far below that, how do I manage that?

That's where the idea of teaching the level of the child comes in. This is providing targeted help for children at various levels. You assess the kids. And it's all the kids in your classroom, and you group them based on their learning level, rather than their age.

And that also allows for some efficiencies because you can organize the kids who are dealing with one particular skill or one particular level in a group and have that group include grade 3, grade 4, grade 5.

So one of the most well-known program organizations that have been implementing this is Pratham, a very talented, highly experienced NGO in India. This sort of approach has also been expanding in government systems at scale. So I just went to see a version of this in Zambia, which in sub-Saharan Africa has been the country to scale up this teaching at the level of the child program most rapidly and has some good evidence for it.

Other countries that have implemented this are Ghana, Cote d'Ivoire, Zambia, as I mentioned. Fundamentally, these two Great Buys align with a third Great Buys, which is providing information on the benefits of education to parents and families to really encourage a system in the sector that there are actually things we can do to improve learning.

And I think what's exciting about these two upgrades is that these are things that are focused on instruction. So I'll put it like this: to improve instruction cost-effectively, you need to improve instruction. There aren't really shortcuts. And I think it's good now that we have evidence of two mechanisms to do that at scale.

Now, just because I've said that these two things are Great Buys, it does not mean that every program that is called structured pedagogy or teaching the level of the child will work. That depends on implementation quality. It depends on really understanding the context of this country. And let me just make a plug for this.

The panel feels strongly that these recommendations are recommendations that start conversations of the country. They are not the end of the conversations of the country. A country needs to decide what the right mechanisms and what the right ingredients of these programs are, and that will differ.

This is just a starting point for policymakers and technical experts in the country to design and decide how best to structure these approaches.

Paulyn Duman: I love that, Ben. Focus on instructions, improving instructions is really important. Just focusing on good teaching. That's the message. Not just, not on the bells and whistles around teaching, but focus on how they can do proper and good teaching to children.

The second aspect of teaching at the level of the child is also important to mention to education leaders. Teachers will need help because they are expected to implement the curriculum at the grade level, but it's not coming from the learner.

The report also highlighted the distinction between cost effectiveness, which is different from effectiveness in global learning.

The spotlight is on learning rather than access. Can you delve into this nuance behind these choices of words?

Benjamin Piper: Some countries really need to be celebrated for the rapid increase in access to education that has happened over the last 20 years. I'm based in Ethiopia. That's one of the examples of the most rapid increases in access to basic education. From the 90s to the 2000s, where it went from 20 percent to 90 percent of kids getting access to basic ed.

The problem has been that getting kids into school isn't enough, as Flan Pritchett says. Schooling ain't learning, right? Just having a kid in school does not mean that they're necessarily learning. They learn if there's good teaching, if they're met at their level. So, as I see it, this issue of learning is the crisis of our day in the sector.

It's encouraging to see that policymakers, development agencies, politicians, and ministers of education have really made a shift to a focus more on learning. I'm also the director of the Center for the Study of Learning. We have a recent study by CGD that just showed how important the issue of learning has been, has increasingly been to policymakers.

So, some people would say that COVID created the learning crisis; actually, it was there well before the pandemic. The pandemic definitely exacerbated it and increased inequalities in many countries.

We have really emphasized in the report the programs that have had a positive effect on learning. Governments in low and middle-income countries are rightly concerned about transitioning rates into secondary school, job creation, youth unemployment, economic prosperity, stability, and growth. These are things that they're worried about, and they should be. But I think one of the things that the evidence shows is that this issue of learning and better foundational learning outcomes is a real strategic pathway toward achieving these goals.

You can't provide technical vocational training for young people that would meet the needs of the job market if the kid can't read, if the young person doesn't have the basic numeracy skills. Many of the challenges that governments are facing in this kind of job creation issue can really be addressed by better learning outcomes. This issue is the one that the panel thinks is important.

It's learning that matters. It's not just schooling.

Paulyn Duman: There are new studies, there are more evidence into what we should prioritize. There are a lot of policies interventions that are being suggested out there.

But what we need to make sure of is that all of these interventions contribute to a coherent education system. Can you describe the characteristics of an education system that is truly oriented to learning?

Benjamin Piper: Yeah, just to say it again, because the panel really wants to emphasize this. These are starting points for conversation. You'll find some interventions that are in each of these categories: the Great Buys, the Good Buys, etc. But all the work really has to happen at the country level, so taking a standalone intervention or project without taking into account the context is just not the solution.

But more importantly, how does this link into the system? How does this connect into the behavior of mid-level civil servants who are busy and now there's one other task that needs to happen? Any change or major reform to improve education must reverberate throughout the system from the policy level, from the minister, approved by the presidency or from the prime minister, and then linked down into the classroom.

And the panel would say that it's important to look at the broader ecosystem for learning and within this, it's crucial to draw the linkages between the political will. Is there actually demand to improve learning? And demand to improve learning at this level? Is there an ability by the system to use evidence on program implementation?

Every single program that I used to be an implementer researcher before taking this job at the Gates Foundation and every program that I was engaged with had problems. There were things that didn't work initially. And some countries and some systems are aligned to respond to those challenges and update and improve and change.

So you need that kind of at the system level. You need to test various approaches. I've seen several countries that have had very successful programs at the end but have had to navigate through implementation research to say, "Okay, let's try this. What's the right package of ed tech that will work best? What's the right package for the teacher that's affordable by the country but also doable? And then how do we actually get that sort of thing done? How do you start thinking about fidelity of implementation, consistency, that a teacher who is used to a certain method of teaching is now asked to be doing something different, and how do they get the support to do that so that it happens on a consistent basis, and how does this link into the broader community and the citizenry, so that you actually are connecting not just to what happens at the school, but happens in the broader civil society.

So one of the things I'm really proud about in this report is that there's a section specifically about implementation and scale and several of the specific sections on various interventions that are included in this version respond on how this should happen from an implementation point of view.

What is the best evidence on and how this works? The report highlights many of these key factors and the policymakers need to view these recommendations within the context of their broader education system.

Regular assessment on whether or not the kids are meeting those benchmarks, and that informs teaching practice. The teachers have structured lesson plans that give them guidance on how to teach, and then they meet on a monthly basis to review the lesson plan and talk about how they're going to make adjustments based on the learning outcomes of their kids.

Training specific on this and, in the Brazilian case, financial investment, incentives provided to municipalities to encourage behavior. And this propelled the state and this particular municipality to move from the bottom to the real top of performers in Brazil's education system, and to be able to stay there over time.

It's an example of kind of the systemic reform that's required to come alongside these instructional interventions. My view is that there's been a lot of kind of innovation on the instructional side at the few classroom levels. That really needs to have the system integration and real work on how to think about aligning this with the broader incentives of the system.

This is an important part of what this panel wants to emphasize is that these are categories of interventions, but it's foolish to think that just saying that now I'm going to do one of these things is going to work unless the broader system is aligned for that improvement and support.

Paulyn Duman: You highlighted the most important thing. It has to be contextualized. The recommendations are a starting point of conversations, and we really need to look at the system's level and see how we can align these innovations, these initiatives to the larger system. Is there political will, is there a demand and how are the different stakeholders also reacting to these recommendations?

So I think that's really important to highlight. People are saying, oh, we have to get the latest technology to personalize learning and to be more effective in providing learning.

We need to incentivize with cash transfers, with food, so that we have the latest technology, and we also have the incentives, the environment for them when they show up in school.

I feel like the report is highlighting that these are not part of the Great Buys, not necessarily. I'm not sure if I am going in the right direction, but can you help us navigate this contradiction?

Benjamin Piper: I think it's worth spending some time on this question because it's an important one. As you said, EdTech fits in the promising category. Look, we're researchers. We're not driven by what we want to be the case. We're driven by what the evidence says. So based on the available evidence as of when we wrote the report and how technology has impacted learning, only two were promising, and we actually highlighted the fact that even in this promising rating, that was based on relatively limited evidence. So let me put it like this.

All too often, technology is deployed without appropriate teacher training, appropriate supporting material. The tech itself is just one part of the category, one part of the solution. It needs to be integrated. And actually, you didn't mention this, but the investment in technology alone, hardware, is actually rated as a Bad Buy.

Consistently it's shown that just buying tablets or buying phones and distributing them doesn't have any impact on learning. It has to be a broader part of the instructional improvement program. That said, the panel does note that there are promising and cost-effective results when existing technologies such as mobile phones or tablets are being used.

Several things happened recently during COVID on this, but in the report, it talks about SMSs via phones to nudge engagement with content, which reduced dropout in Brazil and improved learning in Costa Rica. There are some new studies that have happened in Kenya that are not part of the report that I've seen.

I mentioned it because I'm here in Kenya, on using SMS or phones as part of the classroom has improved learning. A second technology that may have potential and maybe even more so over the last year as AI has increased, which is personalized adaptive learning, adapts that same idea earlier we talked about, putting things at the level of the child. Many of these programs now can adapt to the level of the child. Now, with the caveat that these sorts of approaches that might have success exist where hardware already is in the school. We shouldn't expect hardware to appear from nowhere or hardware alone to do work.

But in both of these cases, we really think the research community needs to work on evidence. So I'm a big believer in that tech again, as in my role as I'm leading the global education team at the Gates Foundation, we have 10 new investments in AI and look forward to finding out what impacts they have on a variety of different categories of instructional improvement programs. That said, we really want to test that out and not just assume. There's more need for research in the AI space. AI, by the way, didn't start with chat GPT. There's been AI in the works for quite a while, but I really personally look forward to increasing evidence and having evidence on the impact of these tools on learning.

But my hunch personally, not as a panelist, but my hunch as a researcher myself, is that most of these interventions will struggle, not with the science of how to improve learning for individual children, but the science of how to do that at scale and how to do that cost-effectively. Having evidence for the panel on cost in this area is going to continue to be critical. I fully expect, on the other hand, that the world of highly effective programs that are working at scale will eventually include, and pretty soon include, increasing amounts of EdTech and AI elements. But the specifics for that will require more evidence. So I think we can see a world where AI can dramatically improve learning as part of a large-scale program, but not every intervention, not every style of AI utilization is going to work. Which is why we need evidence, and we need evidence of costs.

You mentioned the issue of cash transfers. Can I address that as well? So we have to understand that the main goal of this report was to focus on learning outcomes. So there are other goals that countries have policy objectives, poverty reduction, access to education, health behaviors, healthy behaviors. But the evidence on cash transfers shows that we might have many benefits from cash transfers, but very seldom are we seeing direct impacts on learning cost-effectively from cash transfers. Now, it's not the only reason why a country would do a cash transfer. But if you want to know whether or not cash transfers improve learning, the answer at this point is definitely not. Again, read the work of the panel as a starting point for conversations. We would much rather have policymakers review the evidence and decide that for them, even though school feeding isn't necessarily a best buy for learning, it's a priority for us, despite the cost due to government prioritization. But we would much rather have them do the analysis and say, "Yes, we can see that this isn't that cost-effective for learning, but still, we're doing it," and rather than just diving headlong into complex work without having evidence to help them consider as they make these decisions.

Can I just make a caveat? I think the school feeding programs do show evidence of learning, but the question is just not cost-effective as other interventions to improve learning. So the cash transfer doesn't have consistent impacts on learning. School feeding does have impacts on learning, but just not cost-effective compared to other options.

Paulyn Duman: So not on the Great Buys.

Benjamin Piper: yeah, that's why it doesn't make the Great Buys.

Paulyn Duman: Can you also help us understand one of the terms, also in the report, which is the Bad Buys? And what are these seductive traps for government leaders in the world of educational interventions?

Benjamin Piper: This is one of the most important aspects of the report, actually. The "Bad Buys" section talks about interventions where evidence has repeatedly shown that the approaches, as they are typically implemented, are either not effective for learning or not cost-effective for learning. So it might not be politically or proactively realistic to reduce spending on these Bad Buys in some systems, but school systems and leaders should really consider prioritizing more cost-effective interventions.

Investing in certain inputs is key, but this report can actually help you think about how to do that better if you have an emphasis on learning. For example, providing schoolbooks is obviously not bad in and of itself because, of course, kids need books, and of course, the ratio of books to kids needs to be as low as possible.

But providing schoolbooks without training teachers, rolling out a new curriculum that has materials without supporting teachers to implement it, and making sure the content aligns with the curriculum by itself doesn't have a positive impact on learning. That's happened over and over again.

Similarly, sending computers to schools without initial and ongoing training and content that is adjusted to the curriculum may result in these computers just ending up in locked closets rather than enhancing children's learning. These points may seem obvious, but I think it's important to nuance what we're saying.

It is those things alone that aren't improving learning. So how can we adapt how we incorporate the schoolbooks into a structured pedagogy program or link them to a "teaching the right level" intervention? How do we make sure that computers are not seen as a panacea by themselves?

But I think it's an important distinction that the Bad Buys are maybe where to start. We put them at the end, but maybe it's the right place to start.

Paulyn Duman: If you can also provide a road map to government leaders on which interventions would be the consistent landmarks to ensure both effectiveness and cost efficiency in learning that you have been highlighting in this episode what would be that road map for governments?

Benjamin Piper: I would actually go back to what I said earlier, which is these two Great Buys that have been upgraded. One is structured pedagogy and teaching at the right level. It's exciting to see that there's an increasing amount of evidence at scale inside government systems that we are increasingly aware of ways to improve learning outcomes.

That is a really big change for our sector and just can't be emphasized enough. I would love to see an increasing amount of evidence on the particular ingredients, the particular A/B testing. That's when we test two different styles and see what works in that particular context. There's so much more we can learn.

But I think having a starting point of buckets of interventions that can help us is quite exciting. And that's not to leave out the third Great Buy, which is providing information on the benefits of education. It's in some ways very low cost. It's cost-effective, not because it's so effective, but because of how cheap it is. It's a very simple thing that you could add as part of a broader intervention that the country or programs can learn from. But I think in all these things, we, as panelists, are really excited about what we've been able to accomplish with this report but recognize wholeheartedly that the sector needs to grow.

I think 13,000 studies sounds like a lot, but actually when you whittle it down, we only had 400 that we could use for the combination of rigorous and cost-effective. And I think what we've seen recently is that the research community has really responded and is doing more on this. So I'm really hopeful that the evidence will change.

I'm really hopeful that we can update the report and have new reports that can look at specific characteristics because I think the power of having good evidence to inform policymaking is what the rest of the panel is most excited about.

Paulyn Duman: We have listeners who are decision makers, policy makers and technical experts. And they would want to learn more about the studies and about the panel. Can we bring them in the right direction if they want to learn more about the panel and the studies?

Benjamin Piper: Absolutely. So I think the best place to start is with the three reports that I mentioned. The Global Education Evidence Advisory Panel has had two versions of this Smart Buys report in 2020 and 2023. I'd start with the 2023 one, and there is a previous report on COVID and Education. So those are the three reports to start with.

I think one of the things I want to emphasize is that there are a lot of references to very exciting studies in those reports. This is a high-level conversation. In my ideal world, someone would start with that report, bring it to a country, and have conversations at the policy level. At that policy level conversation, "Oh, it's interesting. We should try a teaching at the right level, teaching the level of the child program. Let's get excited about that." But then the smaller technical group, maybe with officers from the Department of Basic Education and Curriculum and technical support from donors, could sit together. But don't just read that section. The section is great, but look at the underlying studies because maybe there's a report from Ghana that's most helpful, or maybe there's something from India that would be most influential. Not only are we wanting this to start the conversation, but do look at the specifics of the research studies to see how this particular context matches what some of the evidence is drawn from.

And we look forward to having more examples of this. I think one of the things that's been exciting is that many ministers have had briefings, conversations, and discussions based on this, and we hope to have more of those. We look forward to having more technical reports on various subcategories of this Smart Buys come out as we move forward.