Data Labs are a new, low-cost way to understand service impact using data that is already collected. Here we look at what Data Labs are and how they can help us identify what is having the greatest impact.

As a society, we are good at sizing our social problems and reporting on them. For example, we know that there are around 250,000 admissions to hospitals for mental illness each year.^[1] But we are not nearly as adept at understanding and reporting on the impact of an intervention in reducing social problems such as mental illness, reoffending or substance abuse.

At the same time, we collect vast amounts of data, especially in government.

It enables comparison of the impact of different types of services and service providers…

A ‘Data Lab’ offers a new, low-cost way to improve our understanding of service impact. It enables comparison of the impact of different types of services and service providers in addressing social problems by using data already collected.

In this article, we explore the concept of a Data Lab, its objectives, how it works, why the model hasn’t happened before, and its potential application in Australia. Throughout the article, we cite the example of the successful Data Lab pilot pioneered in the UK by New Philanthropy Capital (NPC) and the Ministry of Justice (MOJ) in 2013.

What is a Data Lab?

Imagine you are an organisation providing outpatient mental health services. You work hard to deliver your services but are unsure of what happens to your clients. For example, were they readmitted to hospital? And even when you know what happened to your client in key outcome areas such as readmission, you are unsure of the impact of your service on that outcome. Did your service contribute to them staying out of hospital? Or would they have stayed out anyway?

This means a service provider can know what happened to the people that they served.

A Data Lab seeks to answer these questions by generating new insights from existing government data.

Specifically, a Data Lab:

1. Makes client outcomes visible. It provides access to de-identified data on client outcomes that is already collected and held by government. This means a service provider can know what happened to the people that they served. In the UK pilot, service providers access average reoffending rates for their clients (de-identified and aggregated) e.g. 33 per cent of your clients reoffended within one year; on average it took them 48 days to reoffend.
2. Helps to compare the impact of a service against no service. It compares outcomes of the service provider’s clients against a group of similar people. This allows the service provider to learn the difference that its service makes over and above what would have happened anyway (based on a control group). In the UK pilot, the reoffending rates of a provider’s clients are compared with a similar group of individuals. E.g. your clients were 4 percentage points more likely to reoffend.

What are the objectives of a Data Lab?

At the highest level, the objectives of the Data Lab are to

• help improve services and
• fund more of what works.

Data Labs are a low cost way of achieving this by making better use of data that is already collected by government.

Even in situations where a service provider can collect the necessary data, services for evaluating causal impact typically cost around $100-$200k. In contrast, the UK pilot is providing access to client outcomes and a report on comparative impact for approximately $5k per program – a fraction of the cost of bespoke evaluation services.

Tris Lumley, Director of Development at NPC, a key architect of the UK pilot, says that Data Labs provide for “routine production of high quality, low cost evaluations of social policies and interventions”.

A Data Lab has various stakeholders including service providers, government and other funders.

With data access and interpretation, service providers can start conversations and analysis about how their program performed and what improvements need to be made.

To be able to get hold of this sort of evidence in the space of two or three months would have been unthinkable a year ago.

In its first 18 months, the UK Justice Data Lab (JDL)  has provided data access and analysis to 121 service providers. The data access and interpretation has impacted the organisations and their service improvement priorities.

Service provider, Safe Ground’s Business Development Director, Adam Moll, articulated that its participation has “undoubtedly helped us to further validate our theory of change”^[2] and has prompted it to collect more of its own client data. The organisation is excited by the prospect of robust impact data for the first time: “To be able to get hold of this sort of evidence in the space of two or three months would have been unthinkable a year ago.”

The results of the JDL inspired the Prisoner’s Education Trust (PET) to undertake its own analysis (by bringing in a PhD researcher) on what was driving the reoffending results it observed. It is confident in the results. Rod Clark, Chief Executive at PET said, “The JDL is a hugely powerful tool for generating robust evidence on the impact interventions can have on reoffending and I would recommend that those working with prisoners should seriously consider using the service.”

In the UK, in recognition of the JDL’s benefits, the Ministry of Justice has extended the pilot for a second year.

Funders and government may value this evidence enough to build the cost of the Data Lab into their grant making functions.

Funders are better able to see what works for decision-making including the relative impact of different types of services and the effectiveness of different providers of similar services. The Data Lab is also an additional mechanism for making their funding recipients more accountable.

Likewise, if the results are made public the government gets more information on what works to guide grant-making as well as policy development.

The Data Lab may reveal that certain programs and/or service providers are having a negative impact. Funders and government may value this evidence enough to build the cost of the Data Lab into their grant making functions.

How does the Data Lab work?

The mechanics of a Data Lab are simple (see Figure 1) and involves four steps:

1. Service provider submits client names to the Data Lab along with other identifying information.
2. Using data from relevant government department(s), the Data Lab provides de-identified data on clients’ outcomes (aggregated if necessary) to the service provider.
3. The Data Lab provides key statistical analysis services (depending on the service providers’ priorities) including comparing the client group with a matched group.
4. Results about the impact of the services can be made public to generate a body of evidence on the types of services and providers that are most effective in improving client outcomes.

A Data Lab can be run in-house by government or externally by a third party. Likewise, the Data Lab can be funded through grants (from government or philanthropy) or operate on a fee-for-service basis. There are pros and cons to both delivery and funding models.

In-house delivery by government ensures the security and privacy of data and leverages government infrastructure and personnel in the setup phase. Delivery by a third party ensures there is commitment by parties other than government, which should help contribute to sustainability and longevity, and offers greater flexibility to innovate on service offering.

If the Data Lab is funded through grants (or by government), it encourages service provider participation and allows demonstration of the Data Lab’s benefits. The model can progress to a fee-for-service later.

The UK pilot operates in-house in the Ministry of Justice and is providing services free to the sector during the pilot phase. (The Ministry funds the service.)

A similar delivery model could be adopted in Australia, where the government operates the Data Lab in-house during the pilot phase and a third party conducts marketing and sector development. A third party organisation with relationships in the sector and the sector’s trust is better placed to explain the service and its benefits.

Why aren’t we reporting on client outcomes?

There are numerous barriers to accessing and interpreting data on the impact of services. Imagine you provide education services to reduce reoffending. Your current barriers to better understanding impact include: you don’t know what happens to your clients; you are unsure what data would be helpful; you don’t have the internal skills or resources to interpret and manipulate the data; and the cost of hiring a specialist to run tailored statistical analyses is prohibitively expensive (sometimes in the order of $100k-$200k).

The Data Lab attempts to address these barriers by providing access to clients’ outcomes and offering robust statistical analysis at low cost.

The UK pilot has demonstrated success in overcoming some of the barriers associated with data access and interpretation. Two service providers spoke about the discomfort and cost of approaching government or prisons directly for data and how the Data Lab overcame these issues.

Rod Clark, Chief Executive at PET said: “We had been trying to get hold of data on reoffending for a while, but the Justice Data Lab really is the only reliable way to do it.”

Safe Ground’s Adam Moll spoke about the difficulty in talking to prisons and that “having the confidence to say ‘we want this data’ can be quite scary. Partnering with the MOJ makes things a lot easier.”

What are the potential applications in Australia?

There are many areas in which a Data Lab would have value here in Australia especially where government has identified key outcomes that it would like to improve. For example, the NSW 2021 goal areas on crime and offending, preventative health and mental health could benefit from a Data Lab. At the federal level, a Data Lab could operate in employment or disability outcomes.

The primary constraint is whether outcomes data is currently collected at the individual level e.g. reoffending outcomes for Joe Smith, and with sufficient other variables about Joe Smith e.g. date of birth, ethnicity, prior sentence record, and severity of sentence to construct a comparison group of similar individuals. These data conditions don’t exist in all areas but do exist in a few high priority social outcomes areas.

In the UK, NPC is currently exploring, with the relevant government areas, Data Labs in employment, health and education.

What are the implementation challenges?

As with any model, there are implementation challenges.

First, data security and privacy is paramount. These concerns are largely overcome if the Data Lab is operated in house by government, with the government holding the dataset and only providing de-identified and aggregated data on client outcomes to the service provider. The UK opted for this delivery model, in large part to overcome these security and privacy concerns.

Publicising organisations’ results should encourage other service providers to access their results.

Second, there is a question around whether to make the results public or not. Publicising organisations’ results should encourage other service providers to access their results.

It also enables funders to observe the relative impact of services and service providers, which should be useful in discussions about impact and to provide additional information for decision-making.

The arguments against the information being publicly available are also persuasive. A key objective of the Data Lab is to encourage interest and appetite for impact data among service providers, but they may be dissuaded by the public nature of the results. A compromise may be to provide a time lag between providing the results to the service provider and making them public.

Powerful tool

The UK pilot has generated significant excitement in the social policy community. The Data Lab is a powerful tool in the impact measurement landscape and many of the conditions are ripe for the first Australian Data Lab.

SVA with support from its UK partner, NPC, is exploring with a number of government departments possibilities of piloting the Data Lab model here in Australia.

Endnotes

[1] 241,389 admitted patient mental health-related separations in 2012–13, https://mhsa.aihw.gov.au/services/admitted-patient/

[2] http://www.justice.gov.uk/justice-news/safe-ground-positive-relationships-are-central-to-reducing-reoffending