What Can We Really Learn From the 2022 PISA Test Results?

When the international PISA scores were released last fall, the headlines were grim with reports of “tanking math scores” and “fears of mental decline” among American students as well as those around the world. 

PISA, the Program for International Student Assessment, is a worldwide study by the Organization for Economic Cooperation and Development intended to evaluate educational systems by measuring 15-year-old school students’ performance in mathematics, science, and reading. 

But is there really cause to sound the alarms? To find out more, NEA Today spoke to Harry Feder, executive director of the National Center for Fair & Open Testing, also known as FairTest. Since 1985, FairTest has worked to end the misuse and overuse of standardized testing and to ensure that assessment of students, teachers and schools is fair, open, valid, equitable and educationally beneficial. 

After the PISA scores were released, there were many news reports about how math scores dropped dramatically and reading scores stagnated. What is your reaction to the news coverage? 

Harry Feder: The results of PISA 2022 should, like all standardized test results, be filtered through a lens of skepticism about the claims of the test producers and administrators. We must also carefully scrutinize “Chicken Little” claims in the media, which is notorious for manufacturing and hyping education crises. 

Declines in standardized test scores have been the premise for all the failed education reforms of the past forty years, from the publication of “A Nation at Risk” and the No Child Left Behind Act to the charter school movement and now, pushes for universal vouchers and privatization. We must guard against this trap yet again. 

Despite grim headlines about “tanking” math scores, the United States did not do terribly relative to other OECD countries in terms of rankings. U.S. students moved up in rankings for all three subjects (math, reading and science). And in aggregate scores, the United States held its ground in reading and science pretty well from the previous administration while other countries' scores went down

What external factors should be considered when looking at the PISA scores? 

HF: By scrutinizing the performance of the United States versus other OECD  countries, the unsurprising conclusion should be that the PISA test is largely a measure of childhood poverty rates rather than academic achievement.  

The United States leads the OECD in child poverty. Our rate of child poverty is approximately 26%, or higher by some measures. Thus, it is not surprising that as a nation we do not perform as well as most other OECD countries on PISA. If you only compare the percentage of American schools with poverty rates that is equal to those of other OECD countries, the United States does quite well. 

For example, in reading, countries at the top of the PISA list– Slovenia (499), Denmark (489), and Finland (474) have childhood poverty rates below 10%. If one were to measure just the U.S. schools with poverty rates under 10%, our average score would be 562, which would place the U.S. first globally.  

In mathematics, Germany, France and the United Kingdom have child poverty rates between 15 and 18% and have scores of 475, 479 and 489 respectively. If you measured only the US schools with childhood poverty rates of 10-25%, we would score a 508, which would place the U.S. fourth in the OECD. (The U.S. actual score was 465). 

America’s problem on PISA is poverty and inequality, not curriculum and instruction. 

How is an assessment like PISA misleading for the public? 

HF: PISA is a scaled score, norm-referenced, multiple-choice test. Two-thirds of all test takers globally score between 400 and 600 on a section (math, reading and science). Only 2% score over 700. In general, these kinds of tests are set up so results will go down over a longer time frame because as explained by Prof. Andy Hargreaves of Boston College, “once a metric is widely used and has a competitive ranking element, gaming the system leads to overall declines in performance after an early lift, and also has negative side-effects on well-being.”  

So it is not surprising that during the last two decades student performance in mathematics, reading and science all significantly declined in most OECD countries. 

Many point to the COVID pandemic as a cause for lower scores, but PISA pointed to another widespread phenomenon. What was that? 

HF: If one were looking for an actual reason for this decline besides test design and use, the proliferation of technology and handheld device usage by 15-year-olds may very well be the culprit.  

The PISA report offers three reasons to support this. 

First, PISA finds that students who spend less than one hour of “leisure” time on digital devices a day at school scored about 50 points higher in math than students who are on their screens more than five hours a day.  

Despite grim headlines about “tanking” math scores, the United States did not do terribly relative to other OECD countries in terms of rankings. U.S. students moved up in rankings for all three subjects (math, reading and science). And in aggregate scores, the United States held its ground in reading and science pretty well from the previous administration while other countries' scores went down

Second, Andreas Schleicher, the director of the PISA survey, wrote that students who reported feeling distracted by their classmates’ digital habits (65%) scored lower in math.  

Finally, nearly half of students (45%) across the OECD said that they felt “nervous” or “anxious” when they didn’t have their digital devices near them. 

Samuel Abrams, who heads the National Center for the Study of Privatization in Education at Teachers College, Columbia University, attributes Finland’s decline in PISA scores to the introduction of the iPhone and its proliferation of use among Finnish teens over the past several years. 

The OECD is to be commended for attempting to analyze the extent to which creativity and innovation are promoted in national school systems, but the Creativity and Innovation review did not include the United States. The PISA team found it difficult to extract data about the state of creative thinking in American schools because education is delegated by the states to many, often small, local school districts.  

They did cite the work of EL Education, a nonprofit organization with a network of public schools committed to performance-based assessment, as an example of creative and innovative schooling designed to get students to think critically, communicate clearly, and create complex work. 

Overall, the report stated that within the limitations of a snapshot review “it has not been possible to do justice to the rich variety of experiences in schools in the USA.” 

Should the PISA assessment really mean anything for American students and educators?  

HF: Not really, but you wouldn’t know it from the weight policy makers and media ascribe to the results. Researchers have shown that there are no statistically significant relationships between indicators associated with the innovation economy, GDP, adjusted gross income, or purchasing power and PISA.  

International tests do not provide meaningful information about the skills most important for full participation in society in terms of socio-civic and economic outcomes in the G20 countries.  

The information tested on international tests is not the information children will need to compete in the innovation economy and the results do not tell us important information about student proficiency with the vocational competencies necessary to create, innovate, and pursue entrepreneurial opportunities.  

What can be learned from the most recent PISA scores? 

HF: The COVID pandemic caused disruption and trauma for everyone around the world.  

The PISA scores provide some evidence of what we already knew–the pandemic was bad for kids everywhere.