Can an AI tutor help students improve at math? UTM study investigates

As schools contend with rapid developments in artificial intelligence, new research from the University of Toronto Mississauga suggests an AI tutor could help students with math, but only if it holds back answers and coaches them through their problems. 

The findings come amid a global conversation about whether AI helps or harms learning. 

In a field where little research has been done so far, a team led by UTM economics Professor Philip Oreopoulos conducted a rigorous experiment involving more than 6,000 middle school math students in Tennessee to determine under what conditions AI tutors could be beneficial.

“The technology now exists with the potential to make AI an exceptionally good tutor. The challenge, however, is that students want to get through their math assignments quickly and easily,” says Oreopoulos.

“Easier equals less effort and less learning ... The AI tutor has to be designed to teach students how to avoid using the tutor to take the easy route out in solving math problems.” 

To test whether an AI tutor could really help, Oreopoulos collaborated with Michael Liut, an associate professor of mathematical and computational sciences at UTM, to build NUMI, an AI tutor designed to avoid giving direct math solutions and instead provide structured support to students.

In their new study, NUMI was used to teach students how to solve math problems by reviewing past mistakes, while withholding direct solutions.

Students scored higher on a test covering practiced and unpracticed math problems when an AI tutor walked them through their mistakes and then made them demonstrate the same skill correctly three times in a row, compared to students receiving conventional computerized instruction, the study found. 

“As a proof of concept, our study may be the first objective evidence to show that a well-designed AI tutor is more effective than without AI,” says Oreopoulos. 

The support students received from NUMI after making a mistake was designed to promote step-by-step, mathematical reasoning rather than answer-getting, and to increase their overall understanding.

“AI tutoring works by turning students’ mistakes into productive learning moments,” explains Oreopoulos, who published the study with the National Bureau of Economic Research as one of three recent working papers on AI and virtual tutoring.

The results show that students assigned to NUMI gained a modest learning advantage, scoring three points higher than those without an AI tutor. The finding is significant because of growing evidence that AI often harms learning by instantly spitting out answers and bypassing the learning process. 

The next challenge is to build and test an AI tutor with interactive features that will make the learning experience closer to that of engaging with a human tutor, Oreopoulos says. 

“We’re not saying this initial experiment shows AI tutoring will be a game changer that transforms learning,” he cautions. “But the results are exciting because we can now move ahead by adding features to make the tutor more motivating and engaging for students.”

NUMI was constructed with that continuous research purpose in mind, allowing Oreopoulos and his team to switch features on and off experimentally. 

“I would like to test and see what happens when the tutor goes voice to voice,” he says. “The AI tutor could talk in a way that a good human tutor would talk to students – being patient, at times playful and with some humour.”

Developing an AI tutor that truly improves learning could have a transformational impact on students from diverse socioeconomic backgrounds, Oreopoulos says. 

“Well-designed AI tutoring technology offers the potential to scale the tutoring experience to many more students for a much lower cost,” says Oreopoulos, noting that effective but expensive private human tutoring flows mainly to families who need it least.

For parents and teachers, Oreopoulos suggests the relevant question is not whether human or AI tutors are better for student learning.

“In most cases, a good human tutor is better,” he says. “It’s more relevant to compare a well-designed AI tutor with nothing when a student doesn’t understand a problem and no one is available to help them individually.”