As we begin the second year of our partnership with Marnin Studio and reflect on the first year’s progress, one of the areas that has been slower to progress than anticipated has been the development of the Studio’s outcomes measurement framework, systems and practices.

In the interest of sharing and learning, we wanted to explain the process and approach that we have taken, why it has been so challenging and the learning that we have gained along the way, in the hope that it will help (or resonate) with other funders and organisations going through similar journeys.

When we first met Marnin Studio last year they were simply measuring the income women were earning from product sales and the number of women attending the Studio.  Although a good starting point, this didn’t address the deeper outcomes that they were trying to drive for the women and the broader community.  We worked with the Studio to develop a clear program logic model, articulating the end impact and how the activities they were delivering were creating outcomes that could enable this end impact.  As with any organisation progressing on an outcomes measurement journey, this was a critical starting point, without which the Studio wouldn’t have known what to measure or why they were measuring it.

Marnin Studio identified that the end impact that they were driving towards was:

Supporting Aboriginal families and communities to heal previously unaddressed trauma and harm by leveraging their strong relationships and cultural connections to become more empowered and socially / financially included and in doing so, create greater life potential for themselves, their families and their communities.

Marnin Studio’s Program Logic Model

Once this logic model was developed, we were on a roll.  Marnin Studio was thrilled with the clear articulation of their work and started confidently communicating this to funders and partners, securing new awards and winning grants as a result, including the Philanthropy Australia Indigenous Philanthropy Award.  We were thrilled to see them embracing this program logic within their operations and together moved excitedly onto the next step of the process – defining their measurement & evaluation framework.

The framework setting stage is all about developing a roadmap for how measurement and evaluation will be done.  It focusses on defining the specific metrics, systems and processes that will be used for assessing progress on the different outcomes.  Again we completed this quite quickly and with strong enthusiasm from Marnin Studio.

When it came to implementing though, things started to become more challenging.  Metrics that Marnin Studio had assumed would be easy to collect, proved more difficult than expected.  Measurement systems that we needed to implement took a back-seat to sales management systems that were viewed as a greater priority for financial and operational reasons.  Frustrated with the delayed progress, we decided to break the Studio’s measurement framework down and implement in three phases.

Phase 1:

To start we focussed on the metrics that would be easy to implement within the Studio’s existing operations.  This tended to be activity level data (also known as outputs).  We started tracking the number of women engaging with the Studio grouped by demographic (community group, age), as well as by motivation – were they attending as a hobby or as a work-like activity.  We also collected details on workshops that were being held, the different community groups attending and the number of women leading the workshops.

As of July, the Studio was engaging 43 women and increasing this by 2-3 women each month.  Half of the women tend to be in their 20-30s and half pensioners, providing good opportunities for cultural engagement across the two generations.  The women’s motivation for attending the Studio is split evenly with half engaging for work / income opportunities and half for therapeutic purposes. This level of reporting helps to confirm that the Studio is delivering the right types of activities to the right type of women, however this data alone does not provide an indication of the outcomes being achieved.

Phase 2:

To get closer to measuring these outcomes, we worked with the Studio to implement observational assessments of the women who participate in the Studio, measuring changes in skills development, leadership and role modelling.  The Studio is currently, implementing their first round of assessments, which will provide us with baseline data.  Over the next year we will then be reviewing and analysing the women’s behaviours and change over time.

Phase 3:

The final and more exciting stage of the outcomes measurement process is focused on collecting direct inputs from the women and the broader community on the role the Studio is playing in improving their well-being, instilling pride and creating strong community connections.  We are working with the Studio to develop short surveys, which we hope to roll out within the year.  Although, we are eager to move on this as quickly as possible, we also recognise that patience will reward us with meaningful measurement. We are therefore taking a slower and more thoughtful approach to developing these tools, ensuring questions are being asked in the right way.

Over the past year we have often felt frustrated that we were not making enough progress with measuring and reporting Marnin Studio’s impact.  However as we stop and reflect on their starting point and consider their limited resources, we realise that progress is occurring and with a clear plan in place, we are confident that they will continue to progress on this measurement journey.

We are also delighted that the process has enabled them to get clearer about their mission and program logic, and that this in turn is helping then to build a coalition of new supporters and advocates.

We look forward to sharing more updates on their journey in the year ahead.