Robo-marking May Create New Pressures

Short on staff, schools will inevitably look to technology for help and automated grading technologies could soon be rolled out in Australia.

However, without robust national guidelines and an independent advisory body, some schools could be left behind.

“Machine marking” or “automated essay scoring” (AES) is increasingly used internationally, especially in the United States, in a range of high stakes testing contexts where results determine student progression and school funding. The technology is likely to come to Australian schools and could cause controversy without careful governance and consultation with the education community.

A white paper on the issue says AES could potentially alleviate aspects of teachers’ workload but the technology also generates new demands on teachers and administrators, including financial pressure on schools that must procure these proprietary technology packages without advice.

Professor Kalervo Gulson from Sydney Social Sciences and Humanities Advanced Research Centre (SSSHARC) is a researcher in education policy and advances in artificial intelligence. He studies how schools can grapple with and respond to EdTech.

“We know teachers are already experiencing heavy workloads and this new technology could help ease the pressure, so long as the implementation doesn’t create even more work,” said Professor Gulson.

“We don’t recommend automated grading in high stakes testing where the results might impact the student’s future or the school’s funding,” Professor Gulson said. “And like all types of artificial intelligence in schools, it is important that the people impacted by it understand what it will do and have a say in its introduction.”

Co-author Professor Greg Thompson, from Queensland University of Technology, a former high school teacher and an expert in education assessments says, “We should continue to be cautious about any technology that relies on opaqueness being used to make consequential decisions in schools. More time and effort are needed to be spent on opening these systems up to improve both scrutiny and understanding.”

In 2018, the Department of Education, Skills and Employment attempted to implement a form of AES into NAPLAN. However, this “robo-marking” attempt proved politically unpopular and was scrapped by the Minister. The concerns from teachers, teachers’ unions, principals and parents included:

• De-professionalisation of teachers, 
• Inequitable infrastructure in Australian schools, and 
• A lack of transparency from examination authorities as to how marking decisions were made.

The use of artificial intelligence in classrooms can be of great benefit to both teachers and students but it is crucial to have guardrails in place.

“Without political leadership in this area, it is ultimately up to educational institutions and agencies, policymakers, and school communities to work collaboratively to assess the benefits and pitfalls of AES and map the way forward.

“Technology like AES can bring benefits, but our schools need urgent guidance, as well as making sure all schools have the necessary digital infrastructure and human skills,” said Professor Gulson.

Recommendations include:

• Establish uniform, ethical guidelines for schools purchasing AES systems from commercial Edtech companies
• Ensure all schools, including rural and regional, have the necessary digital and human infrastructure to support implementation
• Build transparent best practice guidelines for the use of artificial intelligence in schools
• Initiate an independent advisory body for large-scale assessment in Australia
• Establish a risk-based framework that ranks the potential harm of AES adoption in specific assessment contexts and school settings
• Learn from large-scale rollouts of AES in the US and other jurisdictions, including ethical, legal and financial implications and the impact on school, department and regional decision-making
• Utilise stakeholder expertise across multiple locations and decision-making levels, including educators in schools and classrooms, prior to roll-out.
• Do not use AES in high stakes testing where outcomes are consequential to individuals or to school. 
• If AES is to be widely adopted, Australia must lead the way in the collaborative development of AES guidance, policy, and regulation.

The white paper was developed by the Education Futures Studio and the Education Innovations research program, funded by Sydney Social Sciences and Humanities Advanced Research Centre, a collaborative research facility designed to advance innovative research partnerships and methodologies across the humanities and social sciences.

Image by Andy Barbour