Look up and learn: the role of PISA in education performance

We all remember the nervous anticipation of waiting for an important set of exam results. The anxiety. The sleepless nights. The relief of doing well or the disappointment of hopes unfulfilled.

These sensations form an intrinsic part of the journey to adulthood. Something unavoidable that can hopefully be used as a springboard to future success and opportunity. For most of us, graduation marks the closing of this chapter. Student life and exams are replaced by new pressures and events, both personal and professional. And even those opting for a career in the classroom or academia soon become accustomed to giving out marks, rather than receiving them. Until recently, that is.

The Programme for International Student Assessment (PISA) has started to shine a much-needed light on student outcomes. Created and overseen by the OECD, it compares education systems worldwide by testing the skills and knowledge of 15-year-old students. PISA casts a wide net - more than 70 economies have participated - and this has thrust the impact and performance of different education systems centre stage, enabling policymakers and education leaders to compare, contrast and, if necessary, adjust their approaches.

PISA was created by Andreas Schleicher, Director for Education and Skills at the OECD. Unfortunately, it remains one of too few measurement tools available - a recent OECD study found that only around one in ten of 450 different education reforms put in place between 2008 and 2014 were evaluated for their impact by governments. While this is a trend that Schleicher is seeking to reverse, he readily understands why measuring impact is difficult.

“I think there are several dimensions,” he admits. “One is simply that it is very difficult. The trajectories of good reforms are very long and unfortunately in education there are very few quick fixes. Designing reforms so you can actually build a trajectory and then monitor it and evaluate the reform's success is something that is very hard. Things happen so quickly - you have successive governments taking charge and everyone is interested in their own issues and priorities, rather than the reforms of their predecessor, and so I think that just makes it very difficult to study impact systematically.”

Playing the long game

A German national - although his impeccable English is a testament to both his country's education system and his deep international experience - Schleicher is a physicist by training. His scientific background, in which he was immersed in data and details, is one from which he still draws lessons.

“I started my career in medical research,” he recalls. “When I worked in medical industries and people would tell me that, when you build medical devices, ‘make sure you get it right' because if you get it wrong people would start suing you. But in education we just have a very different culture, with one layer of reform being placed on top of each other and no-one really following up. We still treat education more like an art than a science. We need to build a more evidence-based culture and that, I think, is the real issue.”

The recent OECD report on outcomes highlights how rising education spending and reforms are being hindered by a lack of evaluation, which could hold back their effectiveness and damage educational outcomes. It goes on to say that measuring policy impact more rigorously and consistently will prove more cost-effective in the long run, as well as ensuring that future reforms are built on policies proven to work over a timeframe that is independent of political cycles or pressures.

“If you look at the most successful education systems, looking long-term is the secret of their success,” says Schleicher. “They know where they want to be, they have figured out how to get there and they walk step-by-step very systematically. They get everybody on board - teachers included - and then they build a very strong momentum for reform. Asia is good at this, and some countries in Northern Europe, but it's not widespread.” So, with this in mind, how can organisations like the OECD persuade policymakers to change course? Schleicher believes that it comes down to focusing on developing robust metrics that track successful reforms and that show what the world's leading education systems achieve - which in the education field means the learning outcomes of students.

“Successful reforms are not about making classes smaller,” he says, “or paying teachers more. The question is whether or not such changes are leading to improved learning outcomes. At the OECD, we measure those learning outcomes and track them over time and we tell countries if they are improving or falling behind. And we tell countries whereabouts in the world they can look for answers to their policy challenges. And this has become a very powerful tool of public accountability.” He is also keen to stress that even when countries' education systems differ markedly - as they often do - effective comparisons can nonetheless be made.

“If you want to build your metrics around what each country does then it is very hard, but if you build the metrics around what is achieved, then they have a lot in common,” he continues. “After all, we all want our kids to do well in maths, science, problem-solving, creativity and so on. We could argue for a long time about whether it is better to have more maths hours or more science hours - and this is open to debate - but learning outcomes are goals that are shared across countries, so we need to build our metrics across these variables rather than institutional design.”

PISA: connecting and collaborating

Around 510,000 students in 65 economies took part in the PISA 2012 assessment of reading, mathematics and science, representing about 28 million 15-year-olds globally. When it was introduced in 2000 there was some initial scepticism but it soon proved its value, not least among Schleicher's fellow countrymen and women.

“The common belief among German policymakers was that everything in the education system was fine,” he recalls. “But then they saw the PISA results and it was not just the large performance gap with other countries but there were big social and economic disparities as well. In response, Germany was successful in actually designing reform trajectories with a short, medium and long-term perspective. Germany did well but the East Asia countries did even better. Take Singapore. If you go back to the 1960s, only 2% of its adult population could read or write but they have gone on to build one of the world's best education systems.”

It is clear that PISA's league table style has many benefits - transparency and clarity, for starters. It is easy to see how well a country is ranked and which are performing strongly, or not as the case may be. But Schleicher believes that much depends on the lessons drawn from the league table - looking at it from a narrow country perspective is unhelpful.

“I want people to look at it to identify what the world's most advanced education systems have shown is possible,” he says. “In this way, the PISA league tables can become a very powerful tool for collaboration by bringing countries together. There is much too little of this going on. Part of the problem is that people often believe that education systems are different. They see different cultures and different traditions and think that they can't really learn from each other. If this hurdle cannot be overcome, then we're not going to see the kind of systemic improvements we're after.”

Systemic improvements are just what have happened in Germany since the first PISA results were published. The subsequent changes propelled a stronger academic performance and have seen it rise up the league table, but Schleicher again cites the initial reluctance of policymakers to embrace the findings.

“Two weeks before the results came out I had the opportunity to speak with very senior policymakers there about the findings,” he says. “But they were relaxed and pointed to lots of reports and lots of different priorities that they had to pursue. But then the results made a huge impact in the media. Suddenly, parents were calling politicians and demanding answers and employers were voicing concerns about the country's future competitiveness. All of society kind of rose up and it was this that drove policymakers to do something.”

For Schleicher, this is what public impact is all about. “It comes down to transforming expectations of citizens, showing them the possible horizon and the potential room for improvements,” he says. “I believe these are the keys to actually achieving sustained impact. Impact is not supply-driven - it's really about changing the aspirations of students to learn better, teachers to teach better and schools to become more effective.”

It's clear that this is no overnight fix. However, Schleicher and his colleagues remain determined that the spotlight on learning outcomes will only strengthen in the years to come. With PISA now firmly established in the education sector around the world, few would bet against them.

FURTHER READING

• Maths mission. South Africa's youth face many challenges but they are benefiting from the efforts of Sharanjeet Shan, executive director of Maths Centre, a Johannesburg based not-for-profit
• Class action. Khadijah Abdullah explains how she is spearheading education reform as chief executive of Malaysia's Education and Performance Delivery Unit.
• Leadership lessons. We find out why New York's schools continue to feel the impact of Joel Klein's eight years as chancellor of the city's Department of Education.
• Reading rules. Sir David Bell reflects on a career that has included teaching five-year-olds, inspecting schools, running England's education department and serving as vice-chancellor of the University of Reading.
• Character counts. Getting more young people into employment comes down to the applicant's character, explains Leila Hoteit
• Schools of thought. We speak to a selection of university and college leaders from across the US about measuring the impact of higher education