Online Learning Research: The AI Tutor Trial That Lowered Grades

The most useful piece of online learning research this month is not a success story. In a randomised trial at the University of Maryland, undergraduates whose instructors switched on an AI tutor inside their course site ended the semester with lower grades, and took part much less in the course site itself. For anyone adding an AI assistant to a course, it is the clearest warning yet that the tool can push aside the learning activities that were already working.

What happened

At the end of September 2026, a team led by Jing Liu posted the trial as a working paper through Brown University’s Annenberg Institute (Liu et al., 2026). It ran in the Fall 2025 semester with 2,379 undergraduates and 30 instructors, in courses from 11 of the university’s 12 schools and colleges, including biology, computer science and communication.

The tutor, called the Virtual Study Assistant, was a GPT-4o chatbot built into the university’s learning management system (LMS). It sat in the course’s sidebar, drew on the course’s own materials and declined questions it judged unrelated to the course. Randomisation happened at the instructor level: within each block of comparable courses, some instructors had the tutor switched on for all their students and some didn’t.

Three findings stand out.

Grades went down. In the cleanest comparison, sections of exactly the same course (1,353 students, 22 instructors), students with the tutor scored 4.12 points lower on final grades, or 0.37 standard deviations. Across the full sample the drop was smaller, 2.88 points.

Activity in the course site fell sharply. Recorded participation in the LMS fell by 0.90 standard deviations in the same-course comparison, with page views and active days down too. The authors report declines of similar size in the full sample.

The losses were not shared evenly. In the full sample, first-generation students’ final grades fell by an estimated 5.10 points (0.50 standard deviations), against 2.23 points (0.22 standard deviations) for other students.

The authors put the central finding this way:

What the tutor was actually used for

The usage data are as interesting as the results. Only 14.63 per cent of students with access ever used the tutor, and use was episodic, bunching around exams. When they did use it, they mostly wanted answers:

Practice and test preparation was the second most common kind of request. And the tutor was set up to oblige: it had a tutor mode that guides students towards an answer, but it ran in a direct-instruction mode by default, and the paper reports that only two instructors ended the term with tutor mode switched on. Instructors were given no prescribed way to build the tool into their teaching.

That puts this trial alongside earlier work showing that an assistant which hands over answers can harm later performance, while one designed to make learners do the thinking can help. We covered the best-known of those trials in AI in eLearning: what changed this term.

Two other studies worth reading

Course, chatbot or both? On 13 October 2026, Lendel Narine and John Diaz published a small randomised pilot with US Cooperative Extension professionals learning programme evaluation (Narine & Diaz, 2026). People were assigned to an online certificate course, a custom GPT, both, or neither. The course and the GPT each lifted knowledge test scores to around 91 per cent, against 67 per cent for the control group, and having both added no clear further gain. Confidence in their evaluation skills rose least in the group that had both. The authors call the findings preliminary: only 78 of 144 registrants gave usable data, and attrition differed between groups.

Getting learners to use the tutor at all. In August 2026, Philip Oreopoulos and Nina Low reported a two-year trial of Khanmigo, Khan Academy’s AI tutor, in 18 Tennessee middle schools (Oreopoulos & Low, 2026). The tutor was set up to coach rather than give answers. Maths achievement rose modestly, by about 0.06 to 0.08 standard deviations over a school year, similar to Khan Academy practice without the AI. Almost every student tried it, but the median student messaged it on about a third of practice days, and in only 17 per cent of sessions where they made a mistake. The authors conclude that getting students to use the tutor is the main constraint.

How strong is this evidence?

Treat it as early evidence that should change how you pilot an AI tutor, not as proof that AI tutors lower grades.

The Maryland study is large and randomised, but it is a working paper that hasn’t yet been through peer review. It measured course grades rather than an independent test. Students in both groups could use ChatGPT and Gemini through the university anyway. And the authors are careful about the mechanism: they say the drop in LMS activity doesn’t establish that disengagement caused the lower grades, and with so few students using the tutor, the effect may have worked through broader AI use or changes in instructors’ practice. Fewer page views could even reflect students finding things faster.

What the three studies share is narrower and more useful. Putting an AI tutor in front of learners doesn’t, on its own, make them use it well. Left to their own devices, most ask for answers or don’t ask at all, and a tutor can quietly replace practice they would otherwise have done.

What we’d do

If you’re adding an AI assistant to an online course, or your platform has just switched one on, here’s how we’d respond.

1. Give it a job before you switch it on. Decide which task it supports, such as feedback on a draft answer or hints on a problem set, and tell learners when to use it and when not to. A general helper in the sidebar is what this trial tested.
2. Check the default. If the assistant can give hints before answers, or ask the learner to try first, turn that on. In this trial the answer-first default stayed in place in almost every section.
3. Protect the practice that already works. Quizzes, worked problems and assignments with feedback are where learning happens. If the assistant makes them optional in practice, it is costing you. Our guides to writing multiple-choice questions and effective feedback cover what to keep.
4. Measure before and after. Compare quiz attempts, assignment submissions and final scores for a cohort with the assistant against one without, as in the evaluation phase of the ADDIE model. Watch the learners who were already finding the course hard.
5. Keep an AI-free check. A short closed-book quiz at the start of each module shows what learners can do on their own, whatever tools they used along the way.

What to watch next

The useful follow-ups will be trials that switch the tutor’s default to hint-first, give instructors a plan for using it, and measure learning with an independent test weeks later. Until then, the safest reading of this month’s research is the one the Maryland authors offer themselves: adding AI help to a course changes how learners engage with everything else in it, and that change needs checking.