When AI meets the classroom: some behavioural lessons

On a crisp morning in Brussels, a group of teenagers gathered around their computers with a mix of curiosity and scepticism. Across the continent, in Seville, another classroom buzzed with similar energy. These were not your average school days: the selected 14- to 16-year-olds were about to help JRC scientists understand better how artificial intelligence could shape the way Europe learns.

Why it matters for schools

As European education systems increasingly embrace digital innovation, integrating AI into the classroom holds promise and raises essential questions. This is why we set out to understand how Large Language Model (LLM)-based chatbots can help pupils in mathematics and other subjects, and whether they facilitate long-term learning and critical reflection.

How we did it

We recruited 122 students from two European schools for two experiments. First, we asked students to estimate the value of some coins in a jar. Half of the students were randomly assigned to one of two conditions:

Solution-only: For some of them, the AI chatbot provided its estimate of the value of coins, without any explanation.
Solution + step-by-step explanation: For the rest, it provided the exact estimate along with a step-by-step explanation of how the AI chatbot arrived at that number.

In the second experiment, all students completed a series of school exercises. Half of the students were randomly assigned to one of two different AI chatbot configurations:

“Socratic” AI, which guides students with questions to stimulate deeper thinking.
“Non-Socratic” AI, which offers direct answers without prompting further reflection.

To better understand the impact of these interactions with AI, we collected performance data, interaction logs, and student feedback to evaluate the effectiveness of each approach.

What did we discover?

The results were rather revealing:

Step-by-step explanations help performance

Students who received detailed explanations performed significantly better than those who only got the answer, showcasing the value of transparent, step-by-step learning.

Socratic AI: engaging, but not always effective

Socratic-style chatbots led to richer dialogues and higher interaction. However, they did not result in significantly better test performance, and, surprisingly, some students found them less helpful.

Limited skill retention without AI support

Even if students found AI interactions helpful in completing the exercise, they struggled to apply the learned concepts to new contexts when the chatbot’s help was removed. This raises questions about the long-term effects on learning.

Some lessons for policymakers

These findings are not just about new technologies, but also about the delicate balance between support and independence, between guidance and autonomy for our children.

Encourage explainability, not just results.
AI tools that provide step-by-step reasoning align well with the EU educational goals of evident learning and transparency. They boost immediate academic success, which is an essential metric under Horizon Europe’s digital agenda.
Socrates meets user experience.
Socratic methods can make learning more interactive, but they’re not a cure-all. User experience and adaptability are key; what motivates one student may frustrate another – a factor that should be carefully considered when developing pedagogy.
Complement don’t replace
The limited retention observed suggests that AI chatbots should be companions, not substitutes, for human teachers. The best outcomes arise when technology complements the irreplaceable role of educators.

What’s next for European classrooms?

As the pupils in Brussels and Seville in this experiment logged off, the researchers were left with a clear message: although AI has the potential to transform schools for the better, it can only do so if it is used wisely. To achieve this, we need long-term studies that track not just immediate gains, but lasting understanding and critical thinking.

European education is currently standing at a (digital) crossroads, where the promise, as well as their pitfalls are real. As our findings suggest, we will need to develop AI that explains, engages and helps students become thinkers, not just answer-getters.