Let your robots do the marking?

A short article appearing in the Independent newspaper on April 6th highlights the tensions brewing because of the use of marking software. Anant Agarwal, president of EdX, (Harvard and MIT's non-profit making arm that runs MOOCs), says that the software will be a boon to learning online in the future, because it will allow students to rewrite and resubmit their essays time and again, to improve their grades. He also argues that instant feedback is what today's students crave. What he doesn't say is that it's an essential part of the management of MOOCs, especially if they are regularly enrolling upwards of 100,000 students for each course. How else are they going to assess and mark all those students' work?

This entire approach is reminiscent of the Computer Assisted Learning (CAL) programs of the 1980s, where students worked their way through a linear course of study, interspersed with quizzes and questions to test what they could remember, and a remedial loop to send them back to 'relearn' if they didn't make the grade. The computer marking of that time was simplistic and mainly used for multiple choice questions. All well and good for the 80s, but is it appropriate for today? And even more importantly, are computer software programs actually capable of marking free form essays?

Many academics believe not, and some have even set up an online petition against the use of marking software, claiming that it computers cannot 'read' student essays, are unable to measure the essentials of human communication. They fail, say the protest group, to cope with detecting accuracy, reasoning and critical thinking, adequacy of evidence, ethical issues and stances, convincing arguments, clarity and veracity. So far, around 2000 academics have signed the petition. What do you think?


Photo by Steve Wheeler

Creative Commons License
Let your robots do the marking? by Steve Wheeler is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

Comments

Lenandlar said…
What i think personally is that i wouldn't worry with any course that uses Software to grade my essays. It is that simple for me. However, i am not sure what the general acceptance level will be like. I suspect there will be some uneasiness about the whole thing. This is just, and i say just, the human side of the matter. Can essays be graded by software for context? I have no clue but my own feeling is no, but that is entirely gut-feeling.

As an educator, i would not be happy sending off my students essays to be graded by a machine. Where is my opportunity to learn from my students? Where is my opportunity to provide my insights into their work? Can the computer do that?
Yehoshua Duker said…
From what I understand, the computer marking systems can be "gamed" by those who know what key words the algorithm looks for. I recently used an online tool that claimed to be able to distill the essential points of an essay (similar to these marking programs). I tried it on a paper that I wrote, and it did about at 60% job, with a couple of obvious errors. If we are going to be determining the grades of students based on teasting, we better be sure that the grading is 100% fair and accurate.
Steve Wheeler said…
I think you make a very valid point in your last paragraph, which I concur completely with. I certainly learn from marking my own students' essays, and would hate to miss out on this experience to a machine, even though at times, I admit marking assignments can be a chore!
Steve Wheeler said…
That is indeed an important point Yehoshua - when students are set assignments, marking should be fair and accurate to give them vital feedback on their progress. If it doesn't do so, it's a systemic failure and can actually compound the problem.
Anonymous said…
Like many others, I've enrolled in some MOOCs, then dropped out. The biggest issue for me was the poor quality of feedback.

I've experienced two main types of feedback. Firstly, feedback from computer-marked multiple-guess style assignments, which unsurprisingly was fairly simplistic in nature.

On one MOOC I received peer feedback about short essays. I had high hopes about this, but at best the feedback was unhelpful - such as the review which offered a detailed description of my essay without making any judgement or suggestion whatsoever. Some of the feedback was actively unhelpful. For example, one reviewer berated me for not citing a particular post. Since I knew nothing about this post, obviously I hadn't read it, but I would have been interested in seeing it. Unfortunately my reviewer gave me no clue where to find it.

I suspect that a significant part of the problem is that the assignments that were set on this particular MOOC were very vague, making it very difficult to offer good quality feedback. I signed up for the course in order to improve my writing, but none of the feedback I received offered any indication of how to do this. Unsurprisingly, I ended up deciding not to continue with the course.

I can't see that computer marking of essays would be any better. I think that peer feedback could work for MOOCs - it seems to be the only viable alternative to computer grading. However if people who aren't experienced in giving feedback are going to a good job, then I think the whole process needs to be planned carefully when the course is designed.
Guy Cowley said…
It depends on your perspective on the purpose of education. For many, probably the majority, education is a means to an end. A way of acquiring core skills and a broad enough education to gain the pieces of paper which employers demand as a entry to getting an interview. For them, intellectual curiosity and depth of understanding would be optional extras. Many teachers are 'results machines' not just because that is what their employers demand but also what their customers demand.
Those with intellectual curiosity and love for understanding for its own sake tend to assume that such attributes are necessarily good. Hence the petitions against mechanistic teaching and marking. However they are the Aston Martins in a Ford market. Academics, like any species, will become endangered if they fail to find a fit with the prevailing environment. It is not compulsory to like it!

Popular Posts