Fundamentals Unpacked: randomised controlled trials and experimentation

In medicine and other applied sciences, randomised controlled trials (RCTs) are the primary procedure used to test new protocols before releasing them on a wider scale. The strength of RCTs lies in their experimental design, which enables more accurate reporting of the impact of the treatment and the elimination of several types of bias.

In spite of these advantages, policymakers and analysts have been reluctant to integrate RCTs in policy design processes. This reluctance has been attributed mainly to their supposed cost, as well as to ethical aspects - notably the delays in making a potential solution available to groups in most need.

It can be more difficult to recreate the experimental settings needed to carry out an RCT in a social context than in a science lab, but such criticisms are not really justified. As far as cost is concerned, RCTs can help governments make substantial savings by rejecting policies that don't measure up. And RCTs can stop government implementing ineffective or harmful policies that may damage the very groups they were intended to help.

So, far from being unethical, RCTs actually help policymaking to be more impactful and cost-effective for both government and citizens.

TLA to impact

It's precisely for their inherent efficiency that the “Test, Learn, Adapt” (TLA) method, developed by the Behavioural Insights Team for the UK's Cabinet Office, has embraced RCTs and placed them at the core of policy design.

The key principle behind the TLA method is that policies need to be continuously assessed and improved - which it achieves in two ways. Firstly, TLA endorses an iterative, adaptive learning process, which builds upon past results; secondly, it argues for a strong focus on experimental evidence in policy design.

At CPI, we believe that the TLA approach is a crucial element of policy design, particularly in the use of evidence. This is a vital element of our Public Impact Fundamentals, and achieving impact becomes significantly less likely without it.

However, that does not mean that, in those cases where substantial evidence is lacking, governments should not attempt policy reforms at all. Small- and medium-scale RCTs can prove extremely useful in providing some preliminary evidence before a measure is more widely implemented, and they can significantly improve its chances of success.

The TLA method is explicitly designed to improve policy design. Nevertheless, we argue that this approach can be successfully integrated with the different aspects of the Public Impact Fundamentals, which consider legitimacy and action to be just as important as policy design in contributing to positive impact.

Accordingly, the TLA approach can be used more broadly to enhance all aspects of policymaking, from building support and legitimacy to delivering optimal implementation. For example, we may design an RCT aimed at investigating how to improve aspects of a new initiative, such as the degree of public confidence it enjoys or the level of cooperation between stakeholders.

Learning from the TLA approach

In light of these considerations, we believe that TLA, while providing an excellent methodological basis, should be applied to several elements of the Fundamentals in order to deliver truly successful policies. Focusing exclusively on policy evidence without explicitly considering legitimacy and action may not lead to the desired outcome.

For example, after conducting a multi-pronged RCT aimed at encouraging jobseekers to attend their appointments at an employment centre, we might find that the most efficient incentive is to impose harsh fines on those who repeatedly fail to turn up. The higher the fines, the more likely they are to attend the meeting. In spite of the empirical evidence in its favour, this measure would be unlikely to gain the support of those responsible for its implementation, thus failing to achieve impact.

Overall, we see the relationship between the Public Impact Fundamentals and the TLA approach as mutually reinforcing. While taking into account the importance of experimental results, the Fundamentals are essential for policymakers to go further by integrating other important aspects of effective policymaking - not only evidence but also contextual factors, which are also crucial in ensuring success.

In conclusion, then, we believe that the experimental mindset is key to achieving better policymaking, and by applying it to all three of the Public Impact Fundamentals, governments are likely to strengthen all aspects of their policies.

THE PUBLIC IMPACT FUNDAMENTALS UNPACKED

• Continuing the conversation: Public Impact Fundamentals in focus
• The Public Impact Fundamentals and paradigms of public administration
• The Public Impact Fundamentals and the role of the state