How to measure outcomes
This article is part two of the step-by-step guide to set up an outcomes management approach, and covers how you measure outcomes. See Part one for defining how you create change.
To recap from the first article, outcomes are the positive changes that happen for people as a consequence of an activity. For example, the activity of giving a child a pair of glasses leads to the outcomes of a child being able to see; and the child being able to read in class and gain confidence.
Measuring outcomes helps organisations to test what they are doing, learn whether or not it is working and iterate to have even greater impact. This test, learn and iterate process can improve service design and delivery, drive innovation and support organisations to engage funders and the people they are working with.
How to measure outcomes
Before starting to measure outcomes, you should get clear on what outcomes you are hoping to achieve. Part one can help you to develop a logic model so that you can clearly see what outcomes you are hoping to achieve, and the logic of how your activities will achieve those outcomes.
After developing a logic model, the next step in managing to outcomes is working out how you will measure your outcomes. Measuring your outcomes will help you test whether your activities are creating the change you outlined in your logic model.
Deciding what data to collect
You need to decide how progress towards each outcome will be assessed. The priority outcomes from the logic model need to be translated into measurable elements that can be monitored. You can do this by identifying a set of indicators that will provide data about progress towards the outcome. The selected indicators should have a clear linkage to the desired outcomes, be validated and practical. This data should guide decision-making and inform service delivery (and if it isn’t, you can review the indicators and collect different information).
1. What are indicators?
The outcomes are the changes your program hopes to achieve. To observe whether or not those changes have happened, you need to use indicators. Indicators are the clues that suggest something has, or is going to happen. Think of the petrol gauge on a car. If the petrol gauge is on full, it is a strong suggestion that the petrol tank is full. We use the indicator (gauge), without having to look inside the petrol tank.
When we talk about indicators, we include things that are highly suggestive that a change has happened or will happen (e.g. an indicator that a person is satisfied with a service is that they say they would recommend the service to a friend) and facts that demonstrate that a change has happened (e.g. an indicator that a person is job-ready is that they have been employed for three months). When developing indicators for a program, the indicators should almost always begin with words such as ‘The number of …’ or ‘The percentage of …’ For example one of the indicators from the demonstration table below is: The percentage of parents who spend more time reading to their children.
One or more useful indicators need to be identified for each of the priority outcomes in a logic model. The data collected on those indicators will help show whether change has happened.
Various governments, peak bodies and research institutes have developed libraries of indicators that include example indicators to draw on. A particularly relevant library is the NSW Department of Finance, Services and Innovation’s publicly available Library of Primary Indicators.
2. Considerations when identifying indicators
When identifying indicators, there are a few considerations you should keep in mind:
- The primary focus of any measurement should be learning for the sake of improving. Keeping this in mind can help youto measure the right things so that you can learn what works and improve the effectiveness of your program
- Be practical, there is no need to do everything. To make the job easier, you can:
- start with just one program or geography
- begin by measuring the priority outcomes
- draw on data you are already collecting
- build on existing evidence from other programs, use existing validated measurement tools and/or use libraries of indicators to find relevant indicators.
- utilise the extensive online resources on measuring outcomes.
- Keep in mind the data you already collect, the data collection methods that are available to you and any budget or time constraints you might have.
- Select indicators that are accurate, rigorous and practical. Think about validity, reliability, timing, resourcing, credibility and whether it will be observable.
Data collection tools
Once outcomes and indicators have been defined, the methods for collecting data need to be developed. Outcomes data can be collected through many different methods including surveys and questionnaires, structured and semi-structured interviews, and tailored client assessment tools.
Data collection does not need to be over-complicated. Organisations can utilise surveys they are already running, observations from staff, or data collected by government or other agencies. Technology solutions can assist in the timely collection and analysis of data.
Collating the data
It is important to record how data will be collected and stored. Data should then be compiled in one central repository so that it can be analysed and used. You can start a simple spreadsheet like the one below to keep track of the data you collect.
All new data collection systems should be pilot tested. Treat any data collected in the first round of data collection with caution while you iron out any glitches with your collection methods. Remember also that the process is iterative and your agency or NGO will learn what works best for data collection and which things they want to measure most.
Other resources for outcomes and measurement
The New South Wales Social Innovation Council commissioned SVA to develop a more comprehensive version of this guidance material as part of their work to develop a shared outcomes framework for all providers of human services in New South Wales. They hope to support all providers, including government to work more collaboratively towards a common set of outcomes.
We also produced an indicator library to help providers identify indicators that prove they are achieving certain outcomes. The guidance material helps providers to develop their own logic model with their own outcomes and their own indicators, which may include some of the outcomes and indicators included in the New South Wales Human Services Outcomes Framework. The guidance material and indicator library is available on the Innovation NSW website.
Providers have begun using the guidance material and running their own logic model workshops using the simple tips. One provider reported that running a logic model workshop was a useful exercise for engaging staff in the process of adopting an outcomes focused approach, and also getting more excited about what they do and the impact they are trying to achieve.
Conclusion
Once an organisation has developed a logic model, defined indicators and is using data collection tools to collect data on outcomes, it is set up to manage to outcomes.
The final, and ongoing step is to embed a learning culture (see How culture grows effective outcomes) and use the data that is collected, to inform decision-making to deliver maximum impact. You can read more about each of these steps in the guide: Managing to Outcomes: A guide to developing an outcomes focus.
For more information, contact consulting@socialventures.com.au
