Our current system of measurement in the public sector is not working.
A focus on metrics designed for control can result in gaming, perverse incentives, and ultimately makes the job of public servants harder. For this reason, we need to shift away from using measurement as a means of control, and instead begin using it as a way to learn about complex problems and the people experiencing them, so we can adapt and improve our approach.
In this webinar we explored the opportunities, challenges and practicalities of an alternative approach to measurement - measurement for learning.
Speakers:
Adrian Brown (Executive Director, CPI)
Sara Fernandez (CEO, Oxford Hub)
Professor Jenny Lewis (Professor of Public Policy, School of Social and Political Sciences, The University of Melbourne)
Facilitated by:
Jane-Frances Kelly (Deputy CEO, Education, ANZSOG)
Materials Hub
To take our webinars further, we’ve collated a wide range of material exploring Meaningful Measurement. Scroll down to explore.
The Webinar
Watch the webinar, and read more about what our community is saying.
Check out our wrap-up blog - with key insights and highlights from the webinar, and a graphic summary of the webinar (and bonus time lapse!).
Explore our audience's observations, resources and questions, posted in the webinar group chat.
The Theory
Dive more deeply into this topic by learning more through blogs, articles and reports.
In a series of 3 blogs, CPI’s John Burgoyne sets forth a different measurement approach for public services centred on learning, shaped by CPI’s practical insights and experiences working in local government. The series explores the shift from “measurement for control” to “measurement for learning”, the values and principles required for this shift, and the methods & tools that enable measurement for learning.
How accurate is the phrase, “What gets measured gets managed”? CPI alumnus Danny Buerkli explores.
In “How do we measure public value?” The Mandarin explores what public value is, and summarises literature that explores its measurement.
ANZSOG and the (former) Victorian State Services Authority’s paper, “Performance measurement: Foundations, consequences and futures” examines why the public sector is interested in performance measurement, how it came to be more important, and outlines some of the fundamental assumptions underpinning it.
How does the Australian Government currently view measurement? While there’s not one approach - the Productivity Commission’s Approach to Performance Measurement section of the Report on Government Services 2021 offers insight into existing frameworks and guiding principles around measurement.
Does performance measurement improve public sector performance? A case of Australian government agencies reviews performance measurement systems across three Australian jurisdictions, and finds there is little-to-no impact of performance measurement on improved performance.
While performance measurement is often viewed as apolitical, webinar speaker Professor Jenny Lewis’ article explores the politics and consequences of performance measurement.
Numbers are both powerful and perilous - in her New Yorker article, Hannah Fry explains What Data Can’t Do.
In the video “The Niche and Purpose of Developmental Evaluation”, Michael Quinn Patton shares an introduction to developmental and utilization evaluation, where it comes from and how it can be used.
The most significant change (MSC) technique is a participatory method based on stakeholder narratives. Stakeholders identify what they consider to be the most significant change resulting from the intervention. In a recent webinar recording, Clear Horizon experts Dr Jess Dart and Lauren Siegmann explain MSC, and its impacts. Clear Horizon’s resources page also features a guide to a range of other evaluation methods, complete with tools and further resources.
Professor Megan Williams and Dr Ruth McCausland discuss Connecting Evaluation and Indigenous Knowledges for Country with Practical Action, in an interactive webinar.
The Greater Manchester Combined Authority (GMCA) Research Team (formerly New Economy) has pioneered the development of a cost benefit analysis (CBA) methodology that articulates the fiscal, economic and social value of interventions.
How might we decolonise evaluation? A Kaupapa Māori evaluation lens is described and used to critique international development evaluation in Dr Fiona Cram’s article.
The Indigenous Evaluation Strategy is a whole-of-government framework used by Australian Government agencies to better place Aboriginal and Torres Strait Islander people at the centre of evaluation approaches.
In a powerful TED talk, Dr Phil Goff, explains how measurement has a big role to play in addressing racism and improving policing.
The Equitable Evaluation Framework seeks to shift towards a new measurement paradigm that builds understanding of the systemic and structural barriers that limit possibilities and the ability to thrive.
In Practice
We’ve collated case studies and material to illustrate what these ideas look like in practice, and offer frameworks and tools to support experimentation with a similar approach in your work.
A Developmental Evaluation Primer and A Practitioner's Guide to Developmental Evaluation from the J.W. McConnell Family Foundation outline the basic principles of Developmental Evaluation and provide a range of tools to support its implementation.
Mark Cabaj’s Developmental Evaluation Diagnostic Checklist helps practitioners assess whether developmental evaluation is a good fit for their situations.
Using insight and practical experiences of the Upstream Collaborative, Nesta’s Meaningful Measurement report explores how local government innovators are realigning measurement and evaluation practices to better learn.
Learning pods are one of the internal ‘measurement’ methods that have proven effective at building a learning-oriented culture. The method is detailed in Chris Bolton’s viral blog post on deploying learning and innovation teams in response to COVID-19.
Confirmation Practices are simple routines for systematic reflective practice, developed by Andy Brogan at Easier Inc.
Aotearoa New Zealand’s LSF Dashboard is a measurement tool which informs Treasury’s advice to Ministers on priorities for improving wellbeing.
Through examples of health & social care in Scotland, Dr Ailsa Cook explains how complex problems require complex approaches to measurement. Her position paper - Outcomes Based Approaches in Public Service Reform - is a comprehensive review of literature and current-state practice, combined with practical recommendations for the future.
The Joint Improvement Team’s Talking Points Framework is a personal outcomes approach. It classifies the outcomes important to individuals into three broad categories: quality of life; process; and change.
Old Fire Station has produced a guide to using storytelling to evaluate impact - which details the Storytelling Methodology, and the 7 steps to storytelling.
The Sensemaker software program captures a large number of brief narratives that are interpreted by the people telling the story, using dimensions defined by the implementer. Check out their use cases!
BetterEvaluation is a global community aimed at improving evaluation practice and theory through co-creation, curation, and sharing information. The Australian Evaluation Society is another great community, which offers regular seminars and other resources.
Join us
Our Community of Practice is a forum for conversations over the course of the webinar series, as well as a chance to build new relationships with like-minded people passionate about reimagining government. Content in webinars will be brought to life, with interactive Q&A sessions, problem solving sessions, and reading group discussions.
Register below, and join us as we reimagine government.