From "hitting the target" to "doing the right thing": a paradigm shift for measurement in complex environments
Article highlights
"You can’t do one thing in one place and know that another thing is going to happen elsewhere" @AilsaCook on working in a complex system
Share article@AilsaCook explains how attributing success to intervention, predicting impact & relying on quantitative targets can do more harm than good
Share articleThrough examples of health & social care in Scotland, @AilsaCook explains how complex problems & outcomes require complex approaches
Share articlePartnering for Learning
We put our vision for government into practice through learning partner projects that align with our values and help reimagine government so that it works for everyone.
Focusing on the outcomes that matter
Dr Ailsa Cook is co-founder/director of Matter of Focus, an Edinburgh-based consultancy and software company that helps its public and third sector clients improve outcomes for people that she established in 2017 with Dr Sarah Morton. We discussed her research and practice in the field of public sector outcomes, and I began by asking what had drawn her into this line of work. “In 2004, I was an academic researcher,” she explained, “starting on a two-year project at the University of Glasgow to evidence how partnership working between health and social care affects service users.” The project involved 240 interviews. “We found out that you cannot come up with a single measure for outcomes, because people were talking about the impact the services had on them in so many different ways. We came up with a really tight, well-evidenced framework for understanding the outcomes that are important to service users.
“That led us to work with the Scottish Government, which had launched a joint improvement team to embed the principle of partnership in health and social care. There were about 1,000 people engaged in the programme from service-driven organisations who wanted to become more outcomes-led. And that six-year initiative resulted in the Talking Points personal outcomes approach.” Ailsa then moved to The University of Edinburgh where she led the‘Meaningful and Measurable' research programme. “We got organisations who were all far ahead in outcomes-focused practice to look at the data they were capturing. And it showed that it's never one data point that enables you to say you're making a difference, it's putting it all together.”
Dealing with complexity
Ailsa went on to say that:
We know we're working in a complex system, and it's just not predictable. You can't do one thing in one place and know that another thing is going to happen elsewhere, because there are so many other factors that influence what goes on.
In social care, people are thinking that because they've bought x hours of care for that person, that's the end of the story. But you can't tell anything about the quality of support simply by counting the number of hours delivered. If you open the black box to see what's actually going on, the reality's radically different.
“What's even less well understood,” she added, “is that many important outcomes cannot be measured. You waste time by measuring things that may be important to managers but meaningless to service users. And that's key: what's perceived as a good outcome for an individual may not be what's important to them. So from the external, service point-of-view, I might seem less socially isolated because I've met more people this week, but I might hate them all and feel even lonelier because I'm sat in a room with people I can't relate to.”
What should we infer from this? “People are trying to find the perfect measurement framework when they should be trying to develop good programmes. Staff chase the target, and they reconstruct their service so it fits the target better. And these crass performance management approaches hide all the good stuff. I feel very strongly that we're working with all these amazing organisations, and none of their work is visible at a strategic level. If you're just looking at the outcomes, you miss the story, the bit that really matters - how we got here and why, and what were the brilliant things that all these people brought to the process that enabled us to get here.”
Contributing to positive outcomes
Ailsa suggested how to avoid these mistakes. “In a complex system, we can't attribute outcomes to activities, but we can tell a compelling story about our contribution to outcomes. It's such an important first step, which works at every level of the system from individual practitioners to policymakers. And it opens the door to partnership working. Instead of partnership being a messy thing that gets in the way of our measurement and attribution models, we try to understand our collective contributions. And we're all meant to be around the table for this to happen.”
Her work with the health promotion team provides a telling example. “One of their targets was increasing breastfeeding rates. They'd identified the groups of women most at risk of not breastfeeding or with most barriers to breastfeeding - and were supporting them. But then they're expected to report performance on the basis of total breastfeeding figures in the health board area. And that's not meaningful, because they're not working with every woman in that area.
“So they can tell a very tangible story about their contribution to improving a particular group's rates of breastfeeding, but only a theoretical story about how they contribute to total rates of breastfeeding across the Health Board area. What helps us think about outcomes more meaningfully is breaking down the change process into steps that connect to people's own contributions. It's people who make change. A good outcomes framework reflects the work people actually do - it can open that black box and enable you to break the process down into steps that make sense.”
Promoting personal wellbeing
She went on to highlight the importance of personal contributions to outcomes. “One really exciting piece of work has been with the Health and Social Care Partnership in Midlothian, an area with relatively high levels of unemployment and disadvantage. GPs told us they'd been struggling to work with disadvantaged people in their community, so they introduced wellbeing practitioners into their practices to address the problem.
“Often, the practitioners would just sit and listen to what was going on with their patients. For some, it was quite simple things like wanting to lose weight or reduce anxiety, but many had experienced violence or abuse in the past, and they had complex life histories and conditions, with experience of the justice system, homelessness and addiction.” But after these conversations, they felt valued and respected, because this was the first time they'd ever had a chance to talk about themselves. By using various tools and techniques, the practitioners had helped them move on with their lives. “People gained confidence, they were out doing things that increased their wellbeing, like taking their grandkids to the park, getting therapy, or joining exercise groups.”
As a result, Midlothian's GPs were able to reach 900 people who were finally getting their health issues dealt with. However, the GP practices only had the service for a time-limited period. Funding was running out, and they wanted to make the case for it to continue. “We worked with them to develop an outcome map showing the contributions made by wellbeing service users, how this was very useful to them in improving their lives. And then we mapped all the data and evidence against this outcome map, and we worked with the practitioners to look at the data they'd gathered. And what was brilliant was that they all really enjoyed looking at data that reflected what they did, as opposed to some random things they felt they had no control over. And they were able to produce an evaluation report that showed a tremendous contribution to greater wellbeing.”
The result was very favourable. “On the back of that, the service was refunded and all the GP practices have agreed to contribute to a budget, because they all want a wellbeing practitioner, and three years on they're still committing funding.”
Doing the right thing
Ailsa convinced me of the value of this more qualitative approach to outcomes. “In Scotland, we've got a very explicit public service reform agenda, which is all about coproduction and collaboration.
We've moved through a paradigm where it was about hitting the target. Now we're into this new paradigm - a very welcome one - where it's about doing the right thing. When you're doing the right thing, you do something different every time and it gets a different outcome every time.
So, of course, qualitative data is important, because there's so little quantitative data that can be applied in that way.”
She believes that when you work in a complex system, understanding and learning are also essential. “You can't predict, but you can learn. And in order to learn, you need to understand what's going on. We find that people are enthusiastic about our approach to outcomes, because it helps them to grasp what's actually happening - they can see it makes sense. It's a way to think about their work at all sorts of different levels.”
The positive potential of taking an outcomes approach
Through my conversations with Ailsa and the team at Matter of Focus, I have learned loads about how an outcomes approach can help us address complex problems. While some like to portray the outcomes debate as black and white - you either can measure them or you cannot - their work demonstrates that there is a lot of nuance that is really important to understand. There is a way to measure outcomes that is actually quite harmful. When you try to attribute success to an intervention, claim you can predict impact, and rely exclusively on quantitative targets, you are likely to do more harm than good. We have unfortunately seen a lot of these types of approaches in top-down systems that use numbers as accountability measures, and they gain a lot of criticism for the field of measurement more broadly.
What Ailsa's and Matter of Focus's work demonstrates, though, is that there is massive potential in a complexity conscious approach to outcomes. One that measures for learning, brings collaborators together to understand their contributions, and acknowledges that there is no one standard outcome that can universally quantify impact. As a pioneer in this field, Ailsa has helped countless organizations put their outcomes into action and ultimately create a culture of continuous learning that helps them understand and improve their impact.