Why data and people are the real assets of our new era

I've spent my entire career studying the opportunities and challenges of using new technologies for the public good.

Today's technological landscape, however, is very different from how it was when I started in the early nineties. Back then, the idea of artificial intelligence (AI) was considered science fiction - now it's getting more traction and application. A phone was a phone. No one had heard of “open data” or “big data”; access to the data that did exist was pretty much limited to governments and large organisations, and much of it existed on paper only.

The ever-accelerating rate of technological advances and how they can improve the way we govern is the focus of the GovLab I co-founded four years ago at New York University. Data is in many ways at the heart of the work we do. Clearly, the flood of data we wade in today poses certain new challenges - for example, to privacy and security. But, when leveraged responsibly, data (and its associated analytical methods) provides new opportunities to transform how we govern and how we solve public problems.

Our work is dedicated to minimising the challenges and maximising the opportunities of unlocking data in order to establish fresh insights and innovations, and to the increased potential for distributed knowledge and expertise. It is predicated on the belief that, if harnessed in the right way, technology and in particular the still-emerging field of data science have the potential to usher in a new era of more legitimate and more effective policymaking. You could call it a virtual reimagining of governance.

Tackling the deficits

To understand how technology (and in particular data) can improve governance, we have to take a step back and consider the current situation.

To a large extent, governance today is characterised by a set of deficits. We have a trust deficit, as evidenced by a number of polls and recent election results that suggest a fatigue and scepticism towards old forms of authority. This trust deficit is closely related to what we might call an expertise deficit - the challenges that policymakers have to deal with are far too complex for the levels of expertise that currently exist in government. And we also have an agility deficit, in the sense that government and its processes - which were obviously developed to achieve a certain level of compliance with due process - are often not agile enough to deal with today's fast-moving and ever-evolving challenges.

Within this context of deficits, data science - by which I mean a multidisciplinary blend of data inference, algorithm development and technology that can solve analytically complex problems - provides the means to make policy and government more effective and legitimate. Based on our research, data can improve governance in five key ways.

First, through better situational analysis data science can help policymakers get a more accurate picture of where they and their societies stand. This analysis is instrumental in designing new, more effective solutions to complex problems.

Second, data can help develop better policy and institutional design. There is a lot of talk about user-driven design out there, but equally important is data-driven design, where you actually design for the problems at hand. Only this close symbiotic relationship between problem and solution can actually help us overcome our many challenges.

Third, data and data science can create new knowledge and give a better sense of how society actually works, what causalities exist, and why certain behaviours are being adopted.

Fourth, data can help with prediction. To a surprising degree, many aspects of governance rely on our predictive abilities (think, for example, of economic forecasting). Data can significantly increase our predictive abilities, thus enhancing the responsiveness and effectiveness of governance.

And fifth, data can help with better assessment. It can tell us how policies and programmes have performed, thereby allowing policymakers to iterate and fine-tune their solutions.

Risks aplenty

Clearly, an increased use of data is not risk-free. It's important to understand that data is not a “thing”. Data is a process, where a variety of elements come together, starting from collection to processing to sharing to analysing and ultimately using. At every stage of this chain there are risks involved. For example, at the collection stage you can have “dirty” data being entered, or data being collected in an inappropriate manner. At the processing stage you could have security challenges; at the sharing stage there are risks to privacy and information security.

Much of our research is dedicated to understanding how we can become smarter about tackling these risks. What mechanisms or forms of institutional design can limit potential privacy violations? How can we increase accountability in the way data is used and shared?

It is essential to ask these questions. At the same time, it is equally important to assess the costs of not using data for fear of the risks involved. In some ways, this fear has limited the contemporary conversation and debate. Risks and opportunities exist on a continuum. What we at the GovLab are trying to do is identify the optimal point (or points) on that continuum - all the while recognising that the answer might be different for different societies or contexts, and that the balance between risk and opportunity is not uniform or static.

AI and government

Of course, we can't talk about data science without covering AI. Its biggest advantage is that it allows an organisation - such as a government - to automate certain kinds of processes, particularly around operations. This means that government can become more agile, as well as deliver a more personalised response. It also gives policymakers the opportunity to redirect resources that are currently spent on routine and low-impact activity towards areas that might be in greater need.

This is the value proposition of AI, but it's important to recognise that it plays out differently in different sectors. Nonetheless, whether it's in education, healthcare or security, the essence of AI is about automating certain kinds of activity and processes.

Obviously, one can survive without AI, but there is going to be more and more pressure on governance organisations to start using it. But while I think AI will become essential, I also think that AI without CI - collective intelligence - is a limited undertaking. Without finding new ways to tap into the expertise that already exists in society - whether it is experiential or skills-based expertise - governance institutions will be unable to tackle today's problems meaningfully. It will also hamper AI efforts.

Without access to collective intelligence, AI will not be able to accelerate its (machine) learning to ensure that it provides appropriate and context-sensitive answers. But similarly, without using AI, collective intelligence efforts won't be able to scale, given the huge transaction costs involved in tapping the wisdom of the crowds.

Government is going to have to invest in both AI and CI in order to achieve the legitimacy that is required for effective governing. Every day reminds us of the severity and complexity of challenges we face. I really believe that more effective solutions depend on governance processes that are more accountable, transparent, participatory, and outcomes-driven. We've seen glimmers of how data and people can help with that - a few case studies, pilot projects and experiments around the world. There's a lot more to be done, and I'm excited to be part of the journey.

FURTHER READING

• Briefing Bulletin: Going digital - how governments can use technology to transform lives around the world
• Ever evolving, e-Estonia. As Estonia's prime minster Taavi Roivas, played a pivotal role in the country's technological advancement. We hear how he did it
• Going digital:  how governments can pick up the pace. When it comes to digital government, the gap between rhetoric and reality remains far too wide, says Florian Frey, but it can be closed. Here, he sets out five ways government could improve its digital deployment.
• Unlocking the digital door for developing countries. Although universal access to the internet remains some way off, Hans Kuipers explains what steps can be taken to bridge the enduring digital divide
• Transforming technology, transforming government. Rare is the policymaker who doesn't see digital as a doorway for strengthening public services. But as Miguel Carrasco explains, the pace of the digital evolution means there is always more to do
• Computer says yes. Governments are increasingly reliant on digital technology to deliver public services - and Australia's myGov service is a potential game-changer, says Gary Sterrenberg
• Online, on track? Miguel Carrasco looks at how policymakers can improve the delivery of digital services