Developing a shared measurement framework is easier than you think; moreover it can lead to better outcomes for clients.

Outcomes-based measurement – if you say it fast it doesn’t sound so bad. So why does it feel so hard to introduce, and where do you start?

The need for outcomes-based measurement is often imposed from the outside. It can be driven by external changes, typically a significant reduction in funding, a change in funding requirements and/or structural reform to the sector that is not of the organisation’s own making. Having to adapt and respond to these changes can feel hard.

…how can you start the process in a manageable way that allows you to learn as you go?

This challenge can be made easier if there is interest and support sector-wide for a shared measurement system: a system that enables organisations to understand and measure their impact, not just as individual players, but in collaboration with a number of organisations with a shared mission.

Like our colleagues at New Philanthropy Capital (NPC) in the UK, at SVA Consulting we believe that ‘until we can get more shared measurement into social policy – and especially the charity sector – our efforts to measure what works will be fragmented and we may miss valuable lessons and insights from our peers’^[i].

But how can you start the process in a manageable way that allows you to learn as you go?

One way to approach it is from the bottom up: understanding what is happening from the client’s perspective. By understanding outcomes for individual clients and showing how these changes link back to the services affected by funding changes or reforms, you can provide tangible results that reflect real changes people have experienced in their lives. This initial evidence can help to build momentum and learning around shared measurement, and both drive the conversation forward and provide information that can be refined as you learn from experience. This process also lays the foundation for greater collaboration in service delivery.

This is what is happening in South Australia’s homelessness sector. An initial project by a few organisations to understand the impact of recent sector reforms has been the first step in the whole sector coming to understand the need for, and starting to work its way towards, a shared outcomes-based measurement system.

The process and the conditions for success of this approach are detailed in this article.

South Australia’s homelessness sector

In 2009, the SA Government consolidated the funding for homelessness services to improve service delivery and prompt greater integration. This instigated major structural and cultural change in the way service providers addressed homelessness with a focus on improved access. Since these reforms the funding scenario continues to be uncertain.

…wanted to understand and measure the outcomes for clients resulting from homelessness services, and to understand the effect of the reforms on these outcomes.

Service providers believed that the reforms were having an impact on individual service users, but the sector lacked an evidence-base to demonstrate this impact. Existing data and reporting was limited as the reforms had only recently been implemented and data collection focused on output measures and not client outcomes and impact.

The SA Homelessness Strategy Group (SAHSG) was set up during the reforms to represent homelessness service providers. The SAHSG subcommittee, made up of representatives from Common Ground, Catherine House, Baptist Care and Uniting Communities, wanted to understand and measure the outcomes for clients resulting from homelessness services, and to understand the effect of the reforms on these outcomes. The group also needed to assess the value of the reforms to demonstrate to government, and other social funders, the impact of their services on clients in order to attract continued funding.

In spite of SAHSG’s commitment, they had limited resources to do the evaluation and individually had little time to spend on it given the priorities and needs of their own organisations.

As a result, SAHSG with support from the SA Department for Community & Social Inclusion (DCSI) commissioned SVA Consulting to develop a process to assess the social value of outcomes experienced by individuals as a result of the reforms. The process was to include interviews with individual clients as well as with service providers and caseworkers.

The process of developing shared measures

The process focused on understanding, measuring and valuing the impacts on individuals and linking these back to the reforms.

Figure 1: SROI principles

We decided to base the process on the seven principles of SROI (Social Return on Investment), an internationally recognised process for valuing a program or organisation (see Figure 1). This methodology starts off by understanding the change from the perspective of the stakeholder i.e. the person who has experienced the change. This means that it understands and measures what really matters – the outcomes for individuals rather than abstract indicators of ‘performance’. It is also a methodology which values the outcomes. While this wasn’t an SROI analysis, the principles of SROI were extremely useful for these reasons.

The principles were used to:

• Understand how the reforms have been implemented and how they have had an impact on people using homelessness services.
• Measure: Analyse existing data on outcomes and measure the impact of the reforms on nominated beneficiaries through in-depth interviews.
• Value: Assign values to the outcomes experienced in the case interviews and link these back to the reforms.

Understanding the reforms and measuring success

The best way to show the impact of the reforms was to focus on understanding and measuring the impacts on individual clients.

…interviews with selected clients provided understanding of the changes in outcomes for these individuals over the reform period…

Analysis first defined the logical link from the national reforms (the National Partnership Agreement on Homelessness and the National Affordable Housing Agreement) to changes at state level and then to changes in service delivery on the ground. The DCSI provided support explaining how the system-wide reforms were structured and why. These were expected to link to the outcomes experienced by individual clients.

The reforms had focused on improving access to services. Based on the concept of ‘no wrong door’, every service is seen as a gateway and users only have to tell their story once. This is intended to provide equitable service across target groups and regions, and a standardised quality experience. At the system level there are a number of reform impacts that are already evident to service providers.

Reflecting on the past few years, providers noted that service quality has improved, with more consistent service across regions and target groups, a higher level of collaboration amongst providers, and more accountability through auditing. The reforms also included a major cultural shift from measuring inputs and outputs to measuring outcomes.

Having defined the logic, interviews with selected clients provided understanding of the changes in outcomes for these individuals over the reform period, whether the reforms had contributed to the change, and how reliant the changes were on the reforms. These outcomes were then mapped back to the reforms through discussions with the individuals’ caseworkers. (Figure 2)

Figure 2 Outcomes experienced by individuals

Further analysis grouped the range of outcomes for service users across five life domains that had become apparent as consistent themes for all individuals: health, housing, aspirations, employment/training/income, and connection to society.

The post-reform stories of service users – an Aboriginal woman, a family, a young person, an adult man and a woman who had experienced violence – were selected by agencies as examples of what happens when the system works.

Many reform impacts were experienced in the individual case studies, in particular improved access to supportive housing.

Valuing these outcomes

The final challenge was to identify the value of the change in outcomes that individual clients are experiencing as a result of the reforms, in a way that made sense and did not overclaim the benefits attributable to the reforms.

To get a better understanding about the value of these outcomes to the individuals, indicators were determined and then financial proxies were assigned to represent a market value.

The value of the proxy selected was then reduced to take into account:

• the extent to which things other than the homelessness sector contributed (known as attribution in SROI), and
• the probability of the outcome happening without reforms (known as deadweight).

The result was a realistic view of the value that was created directly by the reforms. Figure 3 works through this process for an example outcome.

Figure 3. Example of value calculation for employment outcome

This analysis resulted in a proxy dollar value for how much the individual was better off in one year.

On average, for each client interviewed the value of the changes in their lives was around $20,000 over one year. As it is likely that value would continue to accrue over the next few years, the total value is likely to be a conservative estimate. Benefits to society as a whole, such as reduced use of public health services, were also not valued in the analysis.

Next steps

Acknowledging that it is still early days, the report has shown that the additional investment in SA’s homelessness sector and the response of this group of service providers has contributed to significant changes in the sector that are starting to have a profound impact on individual outcomes.

By sharing this between organisations and peers and learning how to work together, service providers will be in a better position to improve program delivery…

Significantly, the report identified the need to develop a better understanding of sector-wide outcomes for individual clients and improve information gathering, measurement and evaluation of these outcomes to assess progress.

By sharing this between organisations and peers and learning how to work together, service providers will be in a better position to improve program delivery to deliver better outcomes for individuals. It will also help to develop a consistent message about what success looks like and demonstrate this through meaningful and clear measurement and evaluation of outcomes across the sector.

What the project also demonstrated was that caseworkers on the ground are likely to already understand these outcomes or milestones through their clients’ stories so it is important to engage them in the process of deciding the outcomes. However, this information may not yet be captured, or at least shared. Identifying what to capture and measure can be developed through an iterative process such as this.

With Social Benefit Bonds (SBBs) being mooted by the SA Government and one of the potential pilot areas being homelessness, the sector has even more incentive to develop a robust and plausible system to measure and evaluate individual outcomes.

The report has led to two workshops to kick off the development of a measurement system for the entire sector.

The next stage is to review the process and what is important, involve stakeholders, and refine the original framework allowing the sector to take ownership of it. The refining process will include workshopping all the outcomes, what the indicators are and how to measure them.

As Madge McGuire, Catherine House CEO and a member of the SAHSG subcommittee, said “The process is very accessible and comprehensible for us. Our ideas and experiences have been included along the way, and we’ve continuing to learn from each other as the work evolves. All up it’s been very productive and pain-free.”

The group is learning as it goes, and is on its way to developing shared outcome measures that will both demonstrate the value of its work to funders, but also assist members in building their own capabilities and supporting each other.