A field experiment with nearly 1,000 high school math students in Turkey offers one of the clearest pictures yet of how AI can help in the moment and hurt later.

Researchers Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı and Rei Mariman compared three groups: students with a standard ChatGPT-style assistant ("GPT Base"), students with a tutor version that gave hints and included teacher input ("GPT Tutor"), and a control group with only textbooks and notes. The study was published in the Proceedings of the National Academy of Sciences in 2025.

The results

Group

Practice vs. control

Exam without AI vs. control

GPT Base

48% better

17% worse

GPT Tutor

127% better

No difference

The chat logs explain the gap. Most students using GPT Base simply asked for the answer and copied it, sometimes including its mistakes.

"[I]f humans don't learn, if they start using these tools as a crutch and rely on it, then they won't actually build those fundamental skills," Hamsa Bastani told Knowledge at Wharton.

What it means for buyers

The difference between harm and help came from product design, not from whether AI was present. When evaluating AI tutors, schools should ask vendors:

  • Does the tool give answers by default, or hints?
  • Can teachers shape what it will and won't do?
  • Is there evidence of learning without the tool, not just better performance with it?

Questions and answers

What was the AI study in Turkey?

A randomized controlled trial at a large high school in Turkey during the fall semester of the 2023-24 school year. The researchers ran four 90-minute math review sessions for about fifty 9th-, 10th- and 11th-grade classes, nearly 1,000 students in all. Each class was randomly assigned to GPT Base, GPT Tutor (both built on GPT-4) or textbooks and notes only. Students practiced with those resources, then took an exam without them (PNAS, full text).

Did students who used ChatGPT do better or worse on the exam?

Worse. Students who practiced with GPT Base scored 17% lower on the exam than students who never had AI, a statistically significant drop, even though their practice scores had been 48% higher.

How much did AI raise practice scores?

GPT Base raised practice scores 48% and GPT Tutor 127%, compared with the control group.

What was the difference between GPT Base and GPT Tutor?

GPT Base was designed to mimic the standard ChatGPT interface. GPT Tutor used guardrails designed with teacher input: its prompt included the solution to each problem, instructions to avoid giving away the entire solution, and common student mistakes with matching hints.

Why did students using GPT Base do worse?

They used it as a "crutch," asking for solutions and copying them. The researchers took the most common message students sent GPT Base, "What is the answer?", and asked it for each of the 57 practice problems ten times. It gave a correct answer only 51% of the time.

Did the AI tutor with guardrails improve exam scores?

No. GPT Tutor students' exam scores were statistically indistinguishable from the control group's. The guardrails largely removed the harm but did not add a measurable benefit.

Did students notice the effect?

No. Students who used GPT Base did not perceive that they had performed worse or learned less. Students who used GPT Tutor thought they had done significantly better on the exam, but they had not.

Do the results apply everywhere?

The authors note the limits: two AI tutors, one subject (mathematics) and a single high school in Turkey.

En español: ¿qué encontró el estudio de Turquía sobre la IA en matemáticas?

En un ensayo con casi 1.000 estudiantes de secundaria en Turquía, quienes practicaron con un asistente similar a ChatGPT (GPT Base) mejoraron un 48% en la práctica, pero en el examen sin IA obtuvieron un 17% menos que el grupo de control. Un tutor con salvaguardas (GPT Tutor) mejoró la práctica un 127% y no afectó los resultados del examen.

Sources