The Problems of Peer Assessments at Scale: Cheating, Delays, Inconsistencies
While peer-assessed assignments can be a great learning tool, the credibility of online courses will not endure unless providers take strong action against cheating.
Recently, I took Coursera’s Social Psychology with the Class Central Study Group. The course includes four peer-assessed assignments that are scored by other students taking the course online.
During the Study Group’s bi-weekly meeting, peer assessments were brought up, and we shared some of the experiences we’d had with them, in this course and others. They can be quite frustrating. Here are some of the problems we’ve encountered.
Blank Submissions
While taking a course on academic note-taking, my colleague @manoel was asked to score several assignments that, surprisingly, were blank. Why would anyone submit a blank assignment?
One reason is that some learners will score submissions without even looking at them. For your own score to be released, you must first score other learners’ submissions. So some learners go through the motions just for the sake of getting their own score. As a result, even a blank copy might receive a passing grade.
Another reason is that once a learner has submitted his assignment, he has access to other students’ assignments. He can then copy one of these and resubmit it as his own. Providers like Coursera allow learners to revise and resubmit assignments, which opens the door to widespread plagiarism.
Plagiarism
Sometimes, submitting a blank assignment isn’t even required to access other students’ submissions. A simple Google search does the trick. For instance, Manoel found a website with solutions for all the peer-graded assignments of Coursera’s Social Psychology course we’d just taken. The website also has solutions for other courses, including courses from other platforms.
A few keywords, often taken from the assignment prompt itself, is usually all that’s needed to find solutions online. Some popular courses even have complete walkthroughs on YouTube.
While providers usually ask students to uphold an Honor Code, sometimes by checking a box before submitting an assignment, it doesn’t seem to do much to deter motivated cheaters.
Many learners who spend considerable time and effort on their assignments feel aggrieved that cheaters appear to go unpunished.
Some instructors mention the risk of plagiarism and ask reviewers to actively help catch cheaters by flagging suspect assignments. One way to find if an assignment is plagiarized is to Google a few sentences from an assignment and see if it returns some matches.
Organized Cheating
According to a comment left on Class Central by Coursera mentor Kristy Read, students created WhatsApp groups for some courses and gave each other good grades. Kristy and others feel that the issue mushroomed with Coursera’s free certificates offer during the COVID-19 pandemic.
On YouTube, Manoel also found videos that show how to cheat on popular online courses. These invite learners to join groups on various messaging platforms, such as WhatsApp and Telegram, to progress through courses together, share answers, and give each other passing grades.
Recently, I came across a course discussion in which a mentor said a learner’s assignment was plagiarised. The learner protested, claiming that it was their own work.
Cheating on assignments is not new. The issue was already brought up in 2012 during The Year of the MOOC, when Coursera and other large MOOC providers began to operate. While some measures have been taken, such as disabling the ability to copy/paste other students’ assignments, it appears that over the last nine years, online course providers have made little progress in preventing at-scale cheating on their platforms.
Delays
Delays occur when no one is available to review your assignment. Some course forums are full of pleas to “Please assess my assignment”. In fact, requesting reviews on the forum is one of Coursera’s own recommendations, but not a very useful one considering many course forums are deserted.
In 2020, this issue eased somewhat because of increased learner numbers, but as the surge of interest in online education abates, the requests are reappearing.
In a course I took in 2017, several of my assignments were assessed by the same learners, and I assessed theirs. When few learners are in a particular session of a course, it can sometimes take weeks for results to come back. @ruima waited more than a month for her final assessment in Social Psychology to be reviewed. My wait was shorter — 10 days.
A similar issue can occur when there are no assignments available to review. I have sometimes had to check the course each day for several days before I could review enough assignments to receive my own course grade.
Inconsistent Reviews
Most courses with peer reviews ask learners to review 3 to 5 submissions, then results at the extreme ends are ignored. This helps reduce bias, but it can be confusing for the learner.
Some people will give full points just because the assignment was attempted. Alternatively, some people will give 0 points if they disagree with the assignment’s opinions, even if the rubric asks peers to assess the writing, not the point of view. As a result, some learners who put time and effort into their assignment might get a low grade, while some who merely attempted the assignment might get a passing one.
With participants worldwide, learners have varying command of the language (usually English), which can affect writing assignments as well as reading and commenting on others. Feedback may be minimal, which can be particularly frustrating if points have been deducted without explanation.
Sometimes, such as when there’s a lack of reviewers, your assignment score might end up being determined by a single person. In those cases, whether you get a fair score will entirely depend on that person, which may lead to a lot of frustration.
If you are unhappy with your results, some providers allow you to resubmit your assignment.
Advantages of peer assessments
With so many problems with peer assessments, you might wonder if there is anything good about them. Happily, there are some advantages. By researching, planning, and writing your assignment, you will remember more about the subject. Sometimes, you will find some interesting insights as you assess your peers’ submissions. But don’t copy other assignments and resubmit them as your own!
Your peers might point out issues with your writing that you were unaware of, such as a poorly written essay or arguments that are hard to understand. Use this feedback to learn and improve.
While peer-assessed assignments can be a great learning tool, the credibility of online courses will not endure unless providers take strong action against cheating.
Tags
Ronny De Winter
“Motivated cheaters” = contradictio in terminis 🙂
It went worse over the years. In the early days of moocs the emphasis was more on learning and less on getting certificates.
Steve
Sometimes the motivation is to pass a F/T course (where fees are charged) that use MOOCs as partial credit. The only test appears to be producing a certificate, not demonstrating knowledge of the subject.
Ronny De Winter
Let’s hope this article triggers more the course platforms & providers than the potential cheaters!
Robert Urbaniak
Hi Pat
We sincerely appreciate your engagement with the platform and efforts to make online learning better for everyone. At a time when the world is mostly learning online, we all have the responsibility to make the experience better. We are inspired by our learner community’s passion and commitment to developing new skills by applying themselves across assessments and assignments. However, there are some instances that keep learners from reaching their full potential. Such behavior must be curbed.
We’ve taken note of your observations and excited to share with you and the community that we’re already working on solutions to help address them. Stay tuned.
Thanks
Robert Urbaniak
Sr. Product Manager, Assessments
Coursera
Pat Bowden
Thanks for the feedback, Robert! I’m looking forward to seeing improvements in the peer assessment process.
Mark McPartland
I have similar thoughts regarding Peer Assessments after also taking the Social Physcology Coursera course (great course by the way @Coursera!)
Firstly Peer Assesment is a great part of any MOOC. It just needs to catch up at least to those trying to game it.
You can probably break the students into two major camps
– Those that just want to learn a new skill
– Those that want to learn a new skill, to improve their career
– Those that just want a certificate to use for other purposes (to say they have a skill, compete in the market places for jobs etc)
The first 2 groups are frustrated the 3rd group is able to game the system. Especially the second group which sees their hard work dimished due to the platform integrity being questioned.
Obviously technology exists to check assignments for plagarism, so if not already employed in Coursera courses, should be.
Advising students to seek assignment reviewers on the forums is a poor solution to anything and just faciliates the reviews-for-mates issue noted in the article
One thing that is lacking in online education is sense of community that you get from in-attendance physical learning (and socialising).
It is certainly an area that MOOCs may want to exploit more.
Punishment of cheting on assignments is one thing missing, but so too is reward for being the ‘good student’. For those that participate in forums, grade more than their share of assignments, you can certainly build a community starting around that, where reward may only need to be recognition and being able to contrinute to improved systems.
(Take Google Local Guides as one type of community of the masses. OK a lot more contriutions to work with for Goolge here, but the thing is they pay virtually nothing for this community (give them virtual badges and a chance to attend a conference for free) peanuts, but without it the contributions would likely be a fraction.
Forums in coursera lost a lot of their use when courses when from scheduled weekly releases to on-demand. Can’t have your cake and eat it too, so do need a rethink on how to build community.
Delays in marking do affect motiviation and take up of courses. But the platform can do many things to counter this. Either have SLAs for assignment markings that once breached then a real person from Coursera takes over.
Or utilise your growing community (and advocates) with things like Ubers surge pricing, where greater rewards may be given when there is a demand for marking outstrips supply.
One doesnt want to have a business model that is built on extrinsic motivation, but a combination of intrinsic (the way the platofrm works helps here or how you can contribute) and extrinsic ( Points, Badges, Leaderboards – OK I also took a coursera Gamifcation course about 10 years ago :)) together can help. Nothing is ever cut and dry.
I think the main thing for Coursera and others is to start showing they are doing something in this area , as nothing has been seen happening for a long time
Pat Bowden
Well said, @Mark McPartland!
A student being unfairly graded in Coursera Courses.
It is 2023. I just face the same issues again and again. Peers give me 0 marks for my hard work but in face they won’t have any punishment. Most of them don’t read instructions or guidelines. 9 of 10 of the submitted assignment are in wrong format.
Moreover, it is not fair to non-English students. The peers know your origin soon after they know your name and if your English is native.
Dennis
The issue with delays are too few people take a specialization or course. And peer grading requires three people to grade! I took one on job hunting and only three of students taking the space of one month to grade each other. I am still short of one grader for capstone project. Longer wait entails higher costs to be paid out.
Coursera need to market low enrolled courses aggressively, or change to one person grading or lower the price tag or make some free with certs for more students to take.
Thanks.
Bret
I suspect that delays are deliberate so that you have to pay more – simply to wait for your results…
Genevieve
People don’t review people’s work by the deadline. Also, I have had people review my work 6 months after I had finished the course and already received a certificate. Meanwhile, people currently taking the course were probably waiting for reviews. There has to be a way to fix this. I never understood why people submit blank pages before- now I do.
Bret
I found on Coursera that if you flagged an assignment as plagiarism then they would remove the assignment and you would have to find another assignment to count for your peer reviews.. so instead from then on I mark them as a 0 and comment that it was plagiarism … I found that almost all assignments from the course I was doing were copies – but it didn’t help that the course itself had an “example” that was the answer.. some people just submitted the course’s example.
I also truly hate the “mark two peers assignments for your one submission – it drives most of these behaviours.
ME
Coursera is a cyber diploma mill.
People pay money (or receive financial aid), submit blank, bogus, or copied assignments, and then receive their certificate. They get their sham credential and Coursera gets money.
Coursera revenues and student cheating increased when courses went to continuous sessions with unlimited re-submissions because that drove away instructor involvement and oversight in the courses they created!
Even if Coursera does something about the cheating, that will not make the certificates more credible because instructor involvement does not go beyond the sage on the screen and it’s easy to pass a course without ever watching a video or doing any of the readings, whether you cheat or not.
A month ago I enrolled in a business course and passed without watching any videos or doing any of the readings. I just went straight to the quizzes. With unlimited submissions of auto-graded multiple-choice quizzes and peer-graded assignments, it’s easy to pass and receive a certificate without learning anything, and you don’t even have to cheat to do it.
Another reason for the increase in cheating is the increase in universities accepting Coursera certificates for some sort of credit. The higher the stakes, the more likely students are to cheat. That’s a truth universally known in education. The answer to the problem is human involvement and interaction with students and review of their assignments by qualified instructors, not technology or AI.
It’s a sham at scale.
José Figueiredo
The “education system” based on this model is merely another business where the so called educator outsources their responsibility to assess and mark the assignments!
It is populated mainly by individuals who have understood the trend in the politically correct world of equity in education, where everyone has a right to be equally “qualified” by the so called education system!
Everyone in this “education game” is fully aware of what’s going on and still persist in fooling each other.
Anyone who really wants an education must follow the fundamental methodology of the previously well established education system now made much easier by the access over the internet to gather educational sources.
Real knowledge/education is power and you must work hard to acquire the real education!
selma
I think allowing resubmissions after grading other’s work is causing some of these issues.
In addition, as @Mark McPartland stated, to prevent plagiarism, a program checking similarity of work to previously submitted work during submission could work. Since each course is different, each instructor could determine that similarity percentage so that a submission is not possible if it is a certain percent similar to another work.
Coursera Student
There’s a flip side to the plagiarism checks they do: legitimate assignments can get flagged and it’s a pain to try to address it. I had an assignment flagged and the e-mail said I copied it. Well, given that the assignment was to use a template provided by the instructor (so it had lots of text that everyone leaves in) and recreate figures in the key (which don’t have any weird font sizes or things like that to cause minor deviations), two of the three figures *should* be identical to others’ work. And the sad thing is that on the assignment that was flagged, the third figure contained random data that we were asked to create, and since we all used different random numbers, they don’t all look the same.
At a real university, you could talk with someone about it. The instructor of the course has been responsive, but it sounds like he has no involvement with Coursera’s checks. The only option appears to be to e-mail back to them, and I’m still waiting to hear back from my reply. At a real university, I think something this important would be given a little more attention. But of course there’s a chat feature, where I can chat with someone directly, right? Nope — I’ve been waiting for more than an hour for someone to respond.
I appreciate the work Coursera is doing to stop plagiarism. In another specialization I worked on first, I saw a LOT of plagiarism (such as code that had someone’s name on it, so they didn’t even bother to put their own name!), and I appreciate the ability to flag it. That said, when it flags legitimate assignments and offers severely limited options to try to contact someone about them, that’s pretty disappointing.
Mr. Urbaniak of Coursera commented on this post. Perhaps he could work to address the inability of learners to address unfair claims of plagiarism against them and ensure that the limited avenues of communication are actually available and receive prompt attention.