Two New Studies Reveal a Complex Picture of AI in Education: Helpful Practice or a Hazard to Grades?

If you want a clear glimpse into the cutting-edge research surrounding artificial intelligence in education, look to Jing Liu, director of the Center for Educational Data Science and Innovation at the University of Maryland. Two major studies released by researchers in October offer a striking and highly enlightening contrast regarding how students interact with AI tools. One of the studies hinted that a little AI-powered math practice might help struggling middle schoolers meaningfully improve their foundational skills. The other study, however, found that an integrated AI course assistant actually undermined college students’ engagement with their coursework and ultimately hurt their grades.

Liu wanted the two studies to be released together, he explained, to demonstrate that the integration of artificial intelligence in education is neither “all good” nor “all bad.” The broader academic field is still figuring out how to design, implement, and evaluate tools that genuinely improve learning outcomes over extended periods of time. This complex reality means researchers must study not just the capabilities of the technology itself, but also how students respond to it in practice—including reacting in unexpected ways that its original creators never intended.

In the first of the two experiments, a math practice website and application called Goblins was rolled out to 6,000 middle schoolers in Deer Valley, Arizona, during the 2025–26 academic year. Teachers assigned specific math problems to their students through the platform. An integrated AI tutor then offered step-by-step guidance and help through text chatting, drawing on a virtual whiteboard, and even engaging in spoken conversation. Teachers were strongly encouraged to use the software with struggling students, though some instructors chose to assign it to their entire classes while others skipped the platform altogether.

Students who solved at least two problems a week—amounting to more than 50 completed problems over the course of the school year—scored significantly higher on a year-end standardized assessment in math than similar students who used Goblins less frequently or not at all. Accelerate, an independent, nonprofit research organization that commissioned the study, equated the statistical difference of 0.11 standard deviations to roughly three additional months of learning progress.

However, education experts advise holding off on any premature celebratory music. The study cannot definitively prove that Goblins directly caused the test score gains. Students who used the platform more frequently might simply have been more motivated to practice math in general and might well have tested better in the subject with or without the Goblins application. Accelerate has expressed hope that it can commission a randomized controlled trial in the future to conclusively settle the matter.

Still, one specific result from this middle school study is certainly worth highlighting: Half of the 6,000 participating students actually used Goblins to tackle at least two problems a week, which is the exact usage dosage associated with higher math scores.

That finding is noteworthy because educational technology has long been plagued by a stubborn obstacle that education researchers refer to as the "five-percent problem." A digital learning tool may work beautifully for students who use it precisely as directed, but as a reliable rule of thumb, only about 5 percent of students actually do so. The other 95 percent either fail to log in at home or sit idly through the minutes of scheduled screentime in the classroom. Previous studies on major digital platforms like Khan Academy, IXL, and other instructional software have well documented this persistent challenge of getting children to use their assigned learning tools consistently.

Goblins appeared to successfully reach some students who are notoriously difficult to engage in traditional academic settings, including English language learners and students with disabilities. Liu suspects that this unusual engagement happened because students could talk aloud to a cartoon avatar of their teacher, which was capable of talking back. Moreover, the AI tutor was programmed to speak and understand Spanish as well as six other languages.

It was essentially like talking to your own teacher, Liu noted, adding that this familiar dynamic made the students genuinely want to use the software.

The level of control granted to teachers over assignments may have helped drive engagement as well. Rather than completely surrendering the lesson plan to a computer program, teachers decided which specific problems to assign to which students and retained the option to upload their own customized problems. That dynamic echoes research findings from older generations of educational technology. Another prominent middle school math platform, known as ASSISTments, also produced strong results by relying on relatively small amounts of teacher-assigned practice coupled with instant feedback for students.

Still, researchers caution against overstating how universally engaging or appealing Goblins proved to be. Only 6 percent of the students in this experiment used the platform for at least half an hour a week, which is a more common dosage recommendation in the ed tech industry and for which Goblins’ usage metrics mirror the familiar five-percent problem.

When AI Access Backfires

Liu’s second experiment tested an AI assistant developed right at the University of Maryland. Before deploying the assistant across the entire university to all 40,000 students, Provost Jennifer King Rice asked Liu to study its actual impact first.

During the fall semester of 2025, Liu randomly assigned 15 out of 30 professors to add the AI assistant to their courses. The other 15 professors taught the exact same courses, but without any AI integration. The academic subjects ranged widely from English and math to biology and social sciences. For instance, two professors taught separate sections of the exact same introductory biology course, one utilizing the AI assistant and one operating without it.

Professors supplied the university’s designated AI chatbot with their course syllabuses, required readings, assignment guidelines, and other relevant course materials. Over the course of the semester, students could log in to ask questions about assignment instructions and approaching deadlines or to get help understanding complex academic concepts. The assistant was made available around the clock through Canvas, the university’s official online learning management system.

By the end of the academic year, the 1,100 students enrolled in courses featuring the AI assistant earned lower grades—reduced by an average of 0.11 points on a standard four-point scale, roughly representing the difference between a 3.0, which is a solid B, and a 2.9, which constitutes a low B—compared with the 1,200 students who made up the control group.

Furthermore, these students interacted significantly less with core course materials, such as making fewer comments on the class’s online discussion boards, and were less likely to attend official office hours or meet with their instructors and peers. Notably, these negative effects were even more pronounced for first-generation college students.

Artificial intelligence tools could inadvertently exacerbate existing inequities in learning outcomes, Rice told a University of Maryland publication, emphasizing that this outcome is particularly troubling to university leadership.

Despite the widespread availability of the tool, only 15 percent of students in the AI-enhanced classes ever actually used the assistant. Those who did chose to log in mostly asked the chatbot to provide direct answers rather than taking advantage of its intended tutoring and pedagogical features.

This raised a critical question for researchers: How could a tool used by such a small percentage of students have such broad negative effects across the board?

Liu and Rice suspect that the AI assistant’s constant, visible presence on the course website inadvertently sent a powerful unintended message to students: that the active use of artificial intelligence is broadly encouraged in the class. Consequently, students may have taken that implicit signal as generalized permission to utilize external tools like ChatGPT, Claude, or Gemini outside the university’s official platform to complete their assignments and coursework.

The researchers acknowledge that they cannot definitively establish whether that external outsourcing actually happened. However, if artificial intelligence ended up doing the heavy lifting and the actual work for students, it likely left them learning significantly less over the semester.

The ultimate lesson to be drawn from this research is not that colleges and universities should avoid artificial intelligence altogether, Liu emphasized. Rather, the finding demonstrates that even a carefully designed and well-intentioned educational tool can have wide-ranging consequences far beyond its initial intended use.

University leadership deserves considerable credit for testing the assistant rigorously before launching a full university-wide rollout and for making these troubling, complex results public so that the broader academic and educational technology field can learn valuable lessons from the experiment.