PISA pie in the sky?

The OECD report on PISA 2018 found that over two-thirds of students in the OECD did not fully try on the tests. Eighty per cent of students in Germany did not fully try as did 79 per cent of students in Denmark and Canada and 78 per cent in Switzerland. Some 73 per cent of Australian and New Zealand students did not fully try. In contrast, 46 per cent of Korean students and 60 per cent of Japanese students did not fully try. This variation calls into question the validity of league tables of countries based on PISA results.

Students don’t have much incentive to perform because there are no personal consequences depending on their results. Students and their parents are not given the results and which have no effect on future academic careers at school.

But turning PISA into a high stakes test might be a mistake as it would distract from teaching the curriculum and place additional stress on students and teaching staff.

Further, different groups of students may also have different levels of motivation and may bias comparisons of achievement.

It’s been established that there is a statistically significant positive effect between test-taking effort and test results. Higher motivation and effort led to higher results and low motivation and effort led to lower results.

A study published by the US National Bureau of Economic Research based on data from PISA 2015 found that differences in the proportion of students not fully trying had a large impact on the rankings for several countries. For example, it estimated that Portugal’s ranking in science in PISA 2015 would have improved by 15 places from 31st to 16th if students had fully tried.

Sweden’s ranking would have improved 11 places from 33rd to 22nd and Australia’s ranking by four places from 16th to 12th.

One-third of Australian students are dissatisfied with school which is likely to show up in low motivation and effort in low stakes tests.

The possibility that student effort on PISA has declined helps explain the contradiction between Australia’s PISA and Year 12 results. Some 75-80 per cent of Australian students participating in PISA are in Year 10. The PISA results for these students have declined since 2000 while results for students two years later in Year 12 have improved significantly.

The percentage of the estimated Year 12s that completed Year 12 increased from 68 per cent in 2001 to 79 per cent in 2018. Year 12 assessments are high stakes in comparison to PISA even for less motivated students because they have implications for future education and work opportunities.

Image by Andrea Piacquadio of pexels