Dr Neela Saldanha, Executive Director, Yale Research Initiative on Innovation & Scale (Y-RISE), visited Australia on the invitation of SVA to give the keynote address at our summit on impact at scale. She explained why successful pilots often don’t achieve the same results ‘at scale’ and provided questions to consider when expanding a change program.

View the slides referred to in the podcast.

Dr Saldanha explained that for big problems relating to social change, scale, reach and depth are all used to change social systems.

“It’s about considering issues on both a global scale and in the localised context. And as we consider scaling up, we need to look beyond simply scaling the organisation – often it’s more important to scale the solution,” she said.

“At the Y-RISE, we study the complexities of scaling interventions. What are the differences between impact at scale and impact at a pilot level? As you think about scaling, consider impact at scale,” Dr Saldanha noted.

From pilot to scale up: five questions to consider

Dr Saldanha and the team at Y-RISE encourage people to consider five questions when scaling up a program to achieve impact at scale:

1. If it worked somewhere else, will it work here? It’s easy to get excited about the results of a pilot and forget to consider the contextual elements that made it successful.
2. What could be the unintended consequences of scaling up? It’s important to consider the spill-over effects i.e. ripple effects on those who are not targeted by the intervention. Furthermore, we should understand: who are we displacing?
3. What’s the political calculus of scaling up? Once we start to scale up a program, it gets public attention in a way that a pilot run by a small non-government organisation (NGO) doesn’t. This contextual difference isn’t always considered at concept stage.
4. What if everyone does it? A pilot operates in what economists call a partial equilibrium where only a few variables are analysed, leaving the rest unchanged. Randomised controlled trials (RCTs), for example, compare a treatment group with a control group. Yet the uplift experienced by the treatment group may not be achieved when the broader community experiences the intervention. For example, if we train a small group of entrepreneurs, then they’re hopefully going to be more successful than entrepreneurs who are not trained. They’ll earn more income and have higher consumption. However, if we train every entrepreneur, we level the playing field, and these gains can diminish.
5. What could be the implementation issues? How do we keep implementation fidelity? Often pilots use experienced and focused team members who monitor effects closely – which can be difficult to replicate at scale. Either these high-quality resources aren’t available at scale or are too costly.

Case study: Kenya’s teacher expansion program

Dr Saldanha illustrated how these questions played out in practice for a Kenyan teacher hiring program.¹ In Kenya, as in many countries, teachers are part of the civil service and are in short supply. So, the Kenyan Government implemented a program to expand the number of teachers by appointing short-term contract teachers that would reduce class-sizes and thereby improve classroom learning outcomes.

The study looked at the effectiveness of contract teachers under government management compared to teachers managed by an international NGO.

“What they found was that, consistent with many contract teacher studies, teachers appointed by the NGO did really well: there were learning gains as measured by test scores. But the teachers who were appointed by the government had no effect at all. Same program, same recruitment approach, same pool of people,” Dr Saldanha said.

So what happened? “There were differences in monitoring and implementation – the NGO was monitoring a little more. But the big thing that happened was the political backlash from the expansion of the teacher contract program and expectations of the government.”

“The government had announced a nationwide expansion of the program to 18,000 contract teachers. This provoked resistance from a teachers’ union which demanded civil service protections and union wages for the contract teachers. The contract teachers hired by the government, though not formally in the negotiations, may have had expectations of getting a permanent unionised job, which undermined their current contract and incentives, leading to lower performance.”

“In contrast, the teachers who were hired by the NGO may not have seen themselves as part of the permanent teacher system and so were largely immune to the political struggle between the government and the teachers’ union, so the incentives in the contract worked as expected and the program achieved learning gains. So yes, a contract teacher program works, but who implements it and the political context surrounding its implementation may have implications for the program.

“A very important lesson here is that it matters who’s doing the hiring. It’s not just about trust in government or incentives. It’s also about people’s expectations when a government hires – that’s why a scale up done by government can have different outcomes versus a scale up done by even a large NGO. It’s something that’s not often thought about.”

Scaling it right

When we talk about impact at scale, it’s important to consider a program’s objectives and what we mean by scale.

“What works at scale may be different from scaling what works…”

“Are we talking about a critical mass of the population adopting the program? Or reaching millions across the world with a single program? Do we want to graduate people out of multi-dimensional poverty and generational disadvantage through more complex but holistic programs? Is our aim to change systems? Are we talking about scaled programs that are global and yet highly localised?

“Importantly, as we do this, are we invested in scaling our own organisations or scaling the solution, in which our organisation could play a part? What does it mean for our organisation to be part of a solution? And what could be [the] complexities of things that are run at scale versus in pilot?” Dr Saldanha said.

Quoting author and philanthropist Rohini Nilekani: “‘What works at scale may be different from scaling what works,’” said Dr Saldanha. “We need to think differently about scale. It’s not a linear process.”

Don’t simply take something that succeeds at pilot to more people without considering the barriers to an effective scale up.

About Dr Neela Saldanha

Dr  Neela Saldanha is the Executive Director at the Yale Research Initiative on Innovation and Scale (Y-RISE) at Yale University which focuses on developing the science around scaling policy interventions. Over 15 years in the private sector she led global, scaled teams in marketing, strategy and sales at Nestle, Unilever, Accenture, and PepsiCo.

Impact at scale

SVA, with the support of the AMP Foundation and the Paul Ramsay Foundation, is investigating one of the biggest challenges in the social sector – what it takes to create impact at scale in Australia.

Access a copy of the Impact at Scale paper here.

For more information on the SVA paper on impact at scale please contact Nick Perini on nperini@socialventures.com.au

Notes

1 T Bold, M Kimenyi, G Mwabu, A Ng’ang’a, J Sandefur, ‘Experimental evidence on scaling up education reforms in Kenya’, Journal of Public Economics, Vol 168, 2018, pp 1-20