Redefining rigour: using stories to evaluate systems change?

At the Centre for Public Impact, our work is centred around working towards the wider goal of creating more effective and legitimate governments. This work is not easy. It is slow, and winding, and change does not always occur in the same direction. This is unsurprising, given that we are working within complex and adaptive systems.

One of the key challenges that we face relates to how we measure and evaluate this work. This stems from the unpredictable and tangled nature of complex adaptive systems, which do not lend themselves to traditional methods of measurement or evaluation. Partly this is because change does not happen in expected ways. It often happens in patches or bright spots, and often it’s two steps forward and one back. Capturing this kind of change requires different kinds of methods to those usually used in monitoring and evaluation.

Storytelling is one example of a method that can reveal small changes and bright spots of activity, often at the level of shifts in mental models, values and beliefs that are required for changes within systems. However, because it falls outside of academic orthodoxy, it’s often seen as less valuable or trustworthy and generating less credible evidence. Essentially, we don’t know how to evaluate the storytelling itself to see whether it’s telling us something valuable.

At CPI, we have been exploring Storytelling for Systems Change, and we are aware of others who are exploring how stories can be used to both shape and evaluate system change. However, applying traditional ways of demonstrating rigour don’t fit this approach, particularly if you want to centre equity and the agency and self-determination of the storyteller.

As a researcher who has been taught the importance of rigour in establishing trust in research and evaluation, this has created a bit of a dilemma for me. If the approach doesn’t sit well with usual ways of demonstrating rigorous work, how can I know that I’m doing rigorous work? I’m wondering if criteria exist, in an equitable way that also respects the voice and choices made by the storyteller. And, if we don’t address rigour, what might the consequences be for our work and the credibility of our findings?

Defining rigour

Perhaps this all depends on what definition of rigour we are using. Sage, the global academic publisher defines rigour as:

“...the degree to which research methods are scrupulously and meticulously carried out in order to recognize important influences occurring in the process of conducting the research. It is a set of standards investigators use to evaluate the quality, trustworthiness, and value of research.”

To distil this a bit, it sounds like steps were taken to ensure that this work is as good/reliable/trustworthy as it can be, and therefore we can see it as valuable enough to contribute to understanding and decision-making. To avoid marking our own homework, a set of standards have been created by which we can evaluate this confidently.

These standards within research often focus on demonstrating thought and care, by applying the right method to the right question and using that method in the most objective and transparent way to create credible findings. For research involving numbers, the standards dictate certain tests and approaches to find trends and patterns within the data. Rigour looks like reporting all the patterns you find alongside the reasons you think these patterns might represent something significant and the reasons why the pattern might just occur by chance.

For research involving stories, which is the majority of CPI’s work, the standards of rigorous work are less clear and straightforward. People tell stories from different perspectives and assumptions, and this informs what people think rigour should look like. There are huge arguments in the qualitative evaluation literature about the value of demonstrating rigour, according to these perspectives and assumptions, translated into terms such as trustworthiness, credibility and dependability. Most approaches centre the opinion and the work of the researcher, who takes the analytical decisions, and takes the responsibility to demonstrate they were the right ones. Beyond small scale participatory analysis attempts, none seem to centre equity and the voice and choices of the research subjects.

So, if storytelling looks like the best method, is there a way to create confidence that what we are learning is credible and trustworthy? How do we know this work is as good as it can be?

Purpose and principles

One of the reasons why it’s difficult to apply the established guidelines for creating rigorous work is because they have traditionally been defined by a western academic lens, which discounts more traditional or Indigenous ways of knowing such as storytelling. It’s hard to know if storytelling has been done as “well” as it can be and crucially, it also feels uncomfortable to define this externally, when equity and the authorship of the storyteller are so central to storytelling. Centering authorship is crucial to equity because the perspective of the storyteller can amplify underprivileged voices and because of the important role stories play in trauma and healing.

Instead of defining rigour, if we focus instead on defining its purpose, we may make it easier to avoid using standards and definitions that are often steeped in the values, beliefs and priorities of the dominant academic culture. Focusing on the purpose might help us to define ways of working that create credible and trustworthy claims within a complex adaptive context.

There has been some attempt to do this already. In 2016, FSG proposed four criteria that should be used to reflect on evaluative work to evaluate quality, trustworthiness, and value in complex, adaptive settings:

• Quality of the thinking

• Credibility and legitimacy of the claims

• Cultural responsiveness and context 

• Quality and value of the learning process

Some of these chime, but inevitably some jar too. If the purpose of rigour is to drive something to be as good or reliable as it can be, it feels uncomfortable to look inward for a standard to judge against. The definition and authority should be somehow external, but how might we square this with equity? Who gets to decide?

One solution might be, instead of using these criteria, we look at creating a set of principles to inform external standards that help us judge whether the work is as good as it could be. Principles can be defined by those external to the work, applied by those closer to the work, taking in different settings and different populations that have different norms and needs. By using principles, it might also be possible to define a level of acceptable rigour and still centre equity; avoiding imposing our cultural norms and conflicts which can often undermine other less vocal or privileged groups. 

A different way of understanding rigour for work in complex adaptive systems

So, what might these principles look like? Thinking about this more deeply and in conversation with my creative colleagues, it occurs that transparency and openness are key. There are hints of this in academia, with calls for less definitive claims; rejecting the red line of statistical significance and embracing uncertainty, modesty and openness to being wrong (H/T @Kclarity). We also felt that the concepts of veracity and continual learning and improvement might be central. 

The sense was that it should be less about saying this is “the final truth” because I did the work rigorously. Instead, it should be about presenting some thoughts on the back of some work, based around some principles, and inviting others external to the work to jump in and contribute to the conversation. That might look like extending the ideas, disagreeing, or adding nuance from different contexts or communities that add something to our understanding of how to make things better, and make the work better. It feels like a shift from rigour as a defence mechanism, to rigour as an invitation to collaborate.

In the spirit of this new flavour of rigour, can I invite you in?