Marking homework with the help of AI

Overview and objectives of the trial

Marking papers takes up a significant portion of teachers’ timetables (source: the Ministry): the average is reportedly 6.5 hours per week for humanities subjects. Whilst there are tools available to make this process easier (such as La Quizinière’s online multiple-choice question marking system, or a smartphone app that can identify a pupil’s correct answers on paper or a digital whiteboard simply by taking a photo), the use of artificial intelligence seems even more promising (the Annecy-based company Compilatio is currently trialling software designed for this purpose with volunteer teachers) as it will be able to process larger volumes of text, such as an essay, for example.

But does that mean we should entrust the marking of pupils’ work to a machine?
The aim of this experiment was to use actual work produced by secondary school pupils – taken from official exam papers collected during the 2024 French DNB exam – and to compare human marking with machine marking. Which is faster? Which is more objective? Which is more effective? And finally, is the technology sufficiently advanced to be relied upon? In short, who wins the ‘battle’ between man and machine?

To form as objective an opinion as possible, I obtained seven official exam papers, prior to marking [1]. These were seven papers contained in envelopes from seven different markers who were kind enough to take part in the experiment. I scanned them, page by page, in high resolution (600 dpi) using the photocopier at the examination centre, thereby obtaining PDF files (one file per page).

I then returned the papers to the teachers, and once they had been marked, I scanned them again, this time including the marks awarded, a few annotations and other comments, so that they could be compared at a later date.

Handwriting recognition is much better, but there is still plenty of room for improvement

The first hurdle is that exam papers, like most of pupils’ work, are handwritten. For those familiar with the OCR software [2] of yesteryear, the task at hand seemed daunting: converting a pupil’s handwriting—which is not always legible—into digital text [3]. If the marker has to retype everything, the time saved is obviously nil. But here again, artificial intelligence has enabled fantastic progress. I therefore used NotebookLM, one of the interfaces of the Gemini AI [4], repurposing it from its primary use, which is to process and summarise information contained in documents. It is possible to import different types of documents, and so I fed it a first, uncorrected copy page by page.

Here is the prompt I entered after uploading the document: “Can you convert the entire text into a digital version?”. The result then appears in the interface.

It has to be said that the result looks impressive at first glance. However, on closer inspection, numerous alterations can be spotted, likely occurring when the AI attempts to find coherence where none exists, or simply when handwritten character recognition has been ineffective: the renumbering of questions (omitting the a. or b.), missing accents or errors involving grammatical homophones are corrected, for example (this would be particularly problematic for dictation: in such cases, one must request a faithful reproduction of the student’s text, retaining the errors), but sometimes this goes further in paragraphs that have been poorly ‘read’ by the AI. Thus, in the example below, the student’s use of ‘Firstly’ to introduce their first point likely led the AI to expect a ‘Secondly’ that does not appear. The student actually used ‘next’ to introduce the second argument, which is perfectly acceptable, or at least entirely understandable to a human marker. The AI’s rewriting is therefore no more effective, and is often far worse than the original: this can lead to significant biases in the marking, particularly when entire sentences are rewritten, as is again the case here. In its defence, AI sometimes encounters the same difficulties as the human eye when handwriting is difficult to read and the lines are not spaced out (which should be mandatory on official exam papers). Finally, it sometimes happens that the text ‘read’ bears absolutely no relation to what is written: AI abhors a vacuum and, based on a few fragments of sentences, is capable of inventing anything (this is a form of hallucination [5]).

A paragraph from the student’s essay:

To conclude this section, the human eye remains indispensable, given the sheer number of changes made by AI—whether in its efforts to improve a text or due to its difficulty in recognising handwriting. Furthermore, the time-saving aspect is highly debatable, and may even work against AI, as everything must be double-checked before moving on to the next stage. Nevertheless, we can highlight the real progress made in terms of HTR [6], and with even more sophisticated tools—which are not yet accessible to the general public but do exist—we can hope that this will no longer pose a problem in the near future.

AI-assisted proofreading

After thoroughly proofreading the entire first draft and compiling it into a single PDF document, I used ChatGPT-4o (free version) to correct it.
Here is what I asked it to do:

So I uploaded both documents, and within seconds, the AI provided the initial findings in detail:

That was when I began comparing human proofreading with machine proofreading.

a. Reading comprehension and interpretation skills, and grammar and language skills

Initial findings in terms of marks: there is a difference of just 0.5 marks between the machine’s and the examiner’s marks on the first 11 questions of the first paper (Reading Comprehension and Interpretation Skills, and Grammar and Language Skills). Here are the details of any discrepancies:

Conclusion on all questions: although the difference in marks is minimal, it nevertheless states the obvious. Indeed, the AI ignores the student’s spelling mistakes, which sometimes results in a higher mark. Furthermore, human markers are not infallible and may unintentionally overlook certain errors in the dictation, or forget to mark or grade certain questions in the first part of the exam (though this is very rare across the entire corpus), or even be irritated by the form, which results in the substance being undervalued. Finally, AI is limited to the instructions provided by the marking scheme and lacks the subjectivity—or rather the sensitivity—of a human marker, who is willing to value a particular turn of phrase or the choice of one word over another. The real added value of AI for this part is the extreme speed of marking (a matter of seconds), but this overlooks all the preparatory time required to enable this digital marking.

b. The dictation

As for the dictation, the first draft was marked more harshly by the AI than it would have been by a human marker.
Indeed, whilst all errors were identified, some were miscategorised and others penalised twice. This resulted in a significant 3-point discrepancy between the AI and the human marker. For the other papers, I had to amend my instructions to ensure an accurate assessment.

c. The editorial team

The AI follows the criteria set out in the national marking scheme (originality; organisation and context; expression), and then provides a detailed explanation and justification for its mark. This is an advantage over human markers, who leave very few comments on the paper and even fewer details about the marking scheme used: only the final mark is shown. On the other hand, many markers make their annotations on a separate draft sheet for each essay, or even fill in a table provided by the standardisers, but I do not have these documents.

Here again, there is a noticeable difference in marking between the AI (33/40) and the human marker (38/40), who was likely impressed by the style. Furthermore, from each set of markers’ papers, I selected a paper from the very end of the set: an unconscious comparison of the papers inevitably takes place, and the marker may thus be inclined to mark a good paper more generously if the preceding ones were disappointing.
Out of the entire corpus of seven papers, only one (paper 6) addresses the reflective topic (which is statistically quite consistent with the reality of a standard envelope): I had to specify to the AI that this was a reflective topic, and it applied the marking scheme set out in the national marking scheme. The justification for the marks awarded is once again well-developed and relevant. The difference in marks between the AI and the teacher is minimal (2 marks).

Summary of all answers

Having set out in detail the comparison between human and machine marking on the first exam paper, here is a summary of the results for all seven exam papers:

CONCLUSION

Current trials of AI-based assessment all start with content that is already in digital form, and it is easy to see why: the time wasted on scanning and double-checking the content after it has been read by HTR significantly slows down the processing speed and thus reduces its appeal. However, handwriting remains the norm, particularly for exams.

Once this hurdle has been overcome, the time saved is clear when marking is carried out by AI. However, the generative instructions must be precise, or even refined or modified during the assessment (for example, in the case of a dictation adapted for Year 7), and, above all, it means entrusting a soulless machine with the task of assessing the originality of a pupil’s style. Furthermore, unlike a teacher who can assess specific knowledge and appropriate vocabulary—such as regarding the First World War in the case of this topic—AI does not measure all the skills the pupil has acquired to produce these pieces of work. Moreover, the AI will remain confined to an official marking scheme, which cannot necessarily express all the implicit nuances or even consider every aspect: it merely identifies deviations from the norm. However, across all seven papers, the difference in total marks is only 0.5. Yet this sometimes masks significant discrepancies within a single paper.

Finally, AI often seems more lenient when it comes to the second part of the exam, namely the essay: this is where the difference compared to human markers is most striking. One also senses a certain nervousness on the part of the AI when discrepancies with a human marker are pointed out. Is this a sign of humility, acknowledging human supremacy? In any case, this experiment tends to show that the teacher’s input is truly indispensable and that the AI marking system still has room for improvement. But for how much longer?

Some references:

• BRUYEN, Christian and FIALAIRE, Bernard (rapporteurs), AI and the Future of the Public Service, Senate Report No. 101, 2024–2025.
• DE LA HIGUERA, Colin and IYER, Jotsna, AI for Teachers: An Open Textbook, Artificial Intelligence by and for Teachers, AI4T, 2024.
• KABORÉ André, Assessment in the Context of Overcrowded Classrooms in the Humanities at Joseph Ki-Zerbo University: An Approach Using AMC Software, Proceedings of the Ouagadougou 2022 Conference, Éditions Francophones Universitaires d’Afrique, November 2023.
• MASSE Bastien, AI and assessment, filmed presentation for the “AI Trajectories in AURA” training course, published on 25 June 2024.
• Artificial intelligence in education: research contributions and public policy challenges, January 2024.