A challenging case for refocusing measurement and evaluation around simple measures that drive improvement rather than long-term academic studies that provide ‘proof’.

The non-profit sector is awash with measures that are killing innovation. Driven to ‘prove’ that ‘the program works’, some non-profit organisations are producing an unhelpful mix of reports and academically-controlled studies which can take a long time.

While this ‘proof’ that the program works may help raise money, it does not help innovate quickly. ‘Proving’ a program means taking a static picture and looking at that picture through the eyes of an expert. ‘Improving’ needs a dynamic view which everyone – from the board to frontline workers – can use.

To be truly innovative we need measures that do more than ‘proving that our program works’. We need measures that force us to ask the right questions that will improve our program – quickly.

We’re throwing money at the wrong thing

The non-profit sector wants good measures. Governments around the world are showing interest in the sector, as long as it can ‘prove that it delivers results’. Donors ask non-profit organisations to demonstrate their results and non-profit staff want to know that the work they are doing is making a difference.

The gold standard of ‘proof’ for non-profit organisations is the robust, controlled, academic, longitudinal research study. These types of studies undoubtedly help to raise money and reassure donors that their money is well spent. Sesame Workshop, Big Brothers Big Sisters and Nurse Family Partnership have each shown that their programs work and produce benefits that can be tracked over many years. And the research has helped them raise significant money.

However, this research does not help much in identifying how to improve, nor for that matter in making day-to-day decisions. The desire for ‘proof’ makes it hard to try new ideas and can lead to troubling choices.

It requires simple measures that give quick feedback and allow people to learn swiftly.

Recently, a small education non-profit made the tradeoff, based on a financial assessment, that rather than collect data that would improve its outcomes over the next three years, it would invest in an academic research project that it could leverage to access more funding. In three years’ time, the organisation could be in a better place to secure funding but the program might not have improved in those three years.

For innovation, we need to stop ‘proving’ and start ‘improving’

The crux of the problem is that successful innovation is usually the result of lots of trial and error, not the result of extensive academic reviews. It requires simple measures that give quick feedback and allow people to learn swiftly.

…the ‘build, measure, learn’ feedback loop…  has been slow to seep into the non-profit sector.

Short cycle times are at the heart of most innovative organisations. John Chambers, the CEO of Cisco in its early days, always wanted to put a prototype in front of a customer early: ‘quick and dirty, cheap and easy… then make it better’. This is characterised often as the ‘build, measure, learn’ feedback loop. It is a mentality that has been slow to seep into the non-profit sector.

In Australia, one organisation has taken this approach to heart, in a sector that has been, and still is, entrenched in the opposite approach. One Disease was set up by Sydney-based doctor, entrepreneur and philanthropist, Dr Sam Prince, to systematically eliminate one disease at a time. In 2010, it set out on its first mission – to eliminate scabies in Indigenous communities in Australia where in remote locations, scabies affects 7 out of 10 children before their first birthday. This renders scabies a normal part of childhood for the majority of the community. Although scabies is perceived to be a simple skin infection, its downstream affects are deadly.

The path to eliminating scabies was quite clear – mass drug administration (‘MDA’). The hypothesis was, if you can blanket administer the scabies treatment drug, Ivermectin, to an entire population in a short period of time, you will eliminate scabies. Trials of this type had been proven quite successful in treating conditions such as river blindness in Africa. To this end in 2010, One Disease partnered with a respected academic institution to trial this approach in one Indigenous community with the publicly stated goal of expanding it throughout the north of Australia.

After weeks of gentle conversation, the tent was opened to reveal why.

Concurrently the team at One Disease took the opportunity to gain a greater understanding of scabies and, most importantly, the needs of the community – its ‘customer’. In communities separate to the trial, One Disease realised that some groups in the community presented to the clinic much more often – throwing up some interesting questions:  Why? What was happening in these groups? What was the relationship between the groups?

To get a better understanding, One Disease’s doctor conducted in-depth consultations with community leaders and health workers to collect more ‘data’ of what was happening on the ground.

An observation from one of the communities was that an elder relative had been relegated to a tent behind the house. After weeks of gentle conversation, the tent was opened to reveal why. The elder relative had a hyper-infection of scabies with millions of scabies mites. This relative acted as a ‘core transmitter’ or reservoir of the disease in the family. Each time a child was treated for scabies in the hospital, they went back home and were re-infected by their relative.

It became clear to the One Disease team that scabies was not just ‘one’ disease but that there was a spectrum from ‘simple scabies’ to ‘crusted scabies’ – a far more severe form of scabies and the condition that the elder relative had.

One Disease was in a dilemma. Should it continue down the path that it had promised funders and partners or pivot and shift its primary focus…

Crusted scabies is a highly infectious and devastating condition that occurs when an individual’s immune system is not able to control scabies mite replication. Hyper-infection develops often with up to a million or more scabies mites (compared with 5-10 in simple scabies).

The effect of someone with crusted scabies is far-reaching and without management of the crusted scabies patients, an MDA would be sure to fail.

One Disease was in a dilemma. Should it continue down the path that it had promised funders and partners or pivot and shift its primary focus to the management of crusted scabies patients? While it had a large volume of indicative evidence that an MDA would not work, at least unless other factors were addressed first, it certainly did not have anything resembling academically rigorous ‘proof’. And there would of course be implications in pulling out of the MDA.

Measures that have short cycle times lead to organisations asking the right questions

‘Should we change the program’s direction based on the data?’ is the type of question we need to be asking to have a truly innovative non-profit sector. The questions and resulting decisions can be identified by considering simple, lead-indicator measures, such as the number of clinic presentations for different groups – as in this case.

These types of measures may appear maddeningly simple, or even simplistic. A classic example is standardised school testing, such as the National Assessment Program – Literacy and Numeracy (NAPLAN). As any teacher will tell you, standardised tests are crude, and are not the only measure of how education is contributing to individuals or society at large. But when schools or children get very different test results, it is reasonable to ask ‘why?’.

Innovation means admitting when you’re wrong and being prepared to change direction

One Disease decided that the weight of evidence pointed to the MDA being ineffective without recognition of the role of ‘crusted scabies’ and developed a pioneering program strategy which empowered crusted scabies patients with a chronic care plan. In true trial-and-error fashion, One Disease tested out the approach in conjunction with its inaugural community.

The results were dramatic.

 

Amongst all children living in households with crusted scabies, the average number of days in hospital for scabies declined by 88% in one year compared to the average over the previous three years. Hospital admissions for crusted scabies during the same time period also reduced by 76% compared with the previous year.

One Disease also gained support from the medical community to change the guidelines on how scabies and crusted scabies patients are treated, recognising the importance of using existing health infrastructure to minimise overlap, leverage resources and maximise sustainability of outcomes.

Eighteen months later, the final data out of the MDA trial showed what One Disease had suspected would happen – the trial had not produced any discernable reduction in scabies.

We have to keep the end goal in mind. …what could we do differently that would get us to that end even quicker?

“One of the first thing that funders ask us for is our ‘research framework’,” reflects One Disease CEO, Samantha Cran. “We generally tell them that we have ongoing surveillance (data collection) but we’re an implementation arm, not an academic institute.”

“We have to keep the end goal in mind. The weekly surveillance data allows us to ask the question every week: Is what we’re doing actually helping us to deliver that goal and what could we do differently that would get us to that end even quicker?”.

How do you work out what measures to measure?

In the non-profit sector, the mark of a good innovation measure is the type of conversation its results engender, particularly from people at the frontline where much innovation happens. An acid test of whether a measure helps innovation is the conversation that it stimulates between two field leaders. If it moves from the measure to ‘how did you do that’, it is helping transmit ideas. If the conversation moves to a discussion of context – ‘it’s different in our country’ or it questions the measure itself, it is not helpful.

While the measure … is not perfect, it has helped Habitat for Humanity International ask the right questions and develop better, cheaper ways to house more people.

Habitat for Humanity International measures its impact by ‘families served’. It used to measure ‘homes built’, but it changed as it is easier to house people by renovating existing buildings in places such as Bombay where land is scarce. The change in the measure prompted the US arm of the organisation to examine why it was spending $100k per new home in a land with an oversupply of housing.

While the measure of ‘families served’ is not perfect (some argue that it is quite flawed), it has helped Habitat for Humanity International ask the right questions and develop better, cheaper ways to house more people.

What questions should your measures help you answer?

For your current program and each new innovation or improvement, your measures should help you come to a better understanding of six questions:

1. What’s new? (Does this idea address the issue in a new way?)
2. Why is it good? (What is the impact?)
3. Who does it help now? (Which beneficiaries?)
4. How does it work? (What is the operating model?)
5. How much does it cost? (What are the economics?)
6. Where can it go? (What options are there for growth and scaling?)

Clear answers to all six questions often lets you compare improvement options and can point to where to go next. Unclear answers may show where to look deeper.

For One Disease, the presentation rates at the clinic forced it to question if there was something wrong with the mechanism underpinning its ‘operating model’. A quick feedback loop allowed it to test out another model. For Habitat for Humanity, measuring ‘families served’ helped it realise there was an issue with its economics. Both organisations used simple measures to facilitate quick feedback loops which helped them understand where they could improve.

Why spend resources proving the ‘second best’ option if you know where to improve?

So why is this hard?

None of us like to admit failure, especially to donors. We want public information about our programs to be good. Innovation requires trial and error… and error is failure. Non-profit organisations have a clear incentive – more funding – to produce measures that ‘prove’ with a corresponding disincentive to share measures that ‘improve’.

…many foundations find it hard to admit to error as well.

The reality may be that measuring for improvement is partly a private affair. Show innovations when they are ready for prime time. Show that you are innovating, and not sitting still. Social problems do not solve themselves; they constantly ‘morph’. We need to keep innovating.

Foundations that promote innovation have a role to play too: providing funds for measures of innovation, understanding trial and error (especially error), and applying less pressure to ‘prove’ a static program.

To be fair, many foundations find it hard to admit to error as well. The head of one of the largest foundations in the world, which aims to support innovation, was asked how many of the supported organisations had failed. He replied “None of them!”.

It’s hard for the best of us to improve. But that’s no excuse not to try.

Tell us what you think and your experience of measuring for ‘proving’ and ‘improving’.

About the authors

Jon Huggett is an SVA Consulting advisor and former Partner of Bridgespan non-profit consulting firm in the USA. He is a visiting fellow at the Skoll Centre for Social Entrepreneurship, and founder of The Third Foundation, where he advises social enterprise leaders worldwide.

Prashan Paramanathan is an Associate Consultant with SVA Consulting where he was formerly a Senior Consultant. He is currently the Founding CEO of Australia’s first non-profit crowdfunding start-up ChipIn. He has previously held positions with the World Bank Group starting microfinance banks in rural China as well as with Port Jackson Partners.

If you’d like to know more or share your experience, contact us on consulting@socialventures.com.au