We put our vision for government into practice through learning partner projects that align with our values and help reimagine government so that it works for everyone.
Weapons of math destruction
By Cathy O'Neill
Crown Publishing Group
"Big data", in combination with machine learning algorithms, is increasingly used to replace or augment human decision making. Machines screen résumés, administer personality tests to assess people's aptitude for a job or advise judges on a convict's recidivism risk. If algorithms can process more data, can see patterns that don't reveal themselves to the human eye and don't share our foibles, then what's not to like? A whole lot, as Cathy O'Neil argues in Weapons of Math Destruction.
Despite their veneer of mathematical objectivity these algorithmic machines are anything but. O'Neil argues, convincingly, that if left unchecked predictive policing algorithms will further entrench injustice. Recidivism predictions will discriminate against minorities. Personality tests used to filter applicants for low-paying jobs will screen out individuals with disabilities.
The algorithms we use today are precise enough to be powerful. But they are also flawed enough to do real damage.
The solution, O'Neil proposes, is to regulate these similarly to how credit scores are regulated in the United States. FICO scores are subject to regulatory review, are transparent and have built-in feedback loops which continuously test and improve their predictions. As things stand today, most commercial big data algorithms in use are not regulated, are highly opaque and do not include proper learning mechanisms.
To improve on fairness we will inevitably need to sacrifice some effectiveness. Your neighbours' credit score might indeed be - on average - a good predictor of yours. It is however clearly unfair to penalise you based on your address, yet it is exactly what is happening today. Firms have no incentive to optimise their algorithms for fairness. After all, it is predictive power, not equity, which helps their bottom line. The case for (some form of) regulation seems clear.
O'Neil concedes, however, that the alternative to algorithms - judgment by humans - isn't exactly perfect either. Today, someone might be denied a credit because their e-score is negatively influenced by their neighbours. This is unfair, but in the past a bank teller would have plausibly used a similarly unjust line of reasoning.
This is no excuse for discriminatory algorithms, but we surely need to know what we are comparing them against. In a recent book talk O'Neil conceded that "as bad as it is, I don't know if it [big data] is making things worse or not". Although it feels like an important issue, in much of the literature about algorithmic bias, including her book, it does not receive much attention.
The book also briefly touches upon the enormous potential of using big data and algorithms for the public good. O'Neil mentions a mathematical model she built for New York's Housing and Human Services Department, trying to find out why which services would help homeless families get back into stable housing. We have not even begun to scratch the surface of what is possible once we start to take a serious look at the data governments hold.
For anyone who has been following the debate over algorithms and their discriminatory potential this book will not offer many new arguments. It is, however, a highly readable introduction to the topic and much recommended for anyone interested in gaining a better understanding of these "weapons of math destruction".
