The GENIAL project

How can GENerative AI Tools act a Catalyst for Learning?

Dr Jon Cardoso-Silva & Dr Marcos E. Barreto

LSE Data Science Institute & LSE Department of Statistics

12 Dec 2023

What is GENIAL?

Part I: What are students’ previous experiences with genAI tools?

by Marcos Barreto

The Autumn Term Cohort

Previous Programming Knowledge

Their knowledge of genAI tools

Generative AI tools for learning?

How useful was it?

Part II: How did we gather evidence during the Autumn Term?

by Jon Cardoso-Silva

Timeline

W01 lab

  • Normal lab (no enforcement of ChatGPT)
  • We will be promoting the 📋 Opt-In Survey

W02 lab

  • Normal lab (no enforcement of ChatGPT)
  • We will be promoting the 📋 Opt-In Survey

W03 lab

  • 1st Half: Normal lab
  • 2nd Half: Work independently using ChatGPT actively
    (no tips, no guidance, no instructions)

W04 lab

  • 1st Half: Normal lab
  • 2nd Half: Work independently using ChatGPT actively
    (no tips, no guidance, no instructions)

W05 lab

  • Normal lab (genAI tools are neither enforced nor barred)
  • Students complete a brief survey at the end of the session.

W07 lab

  • 1st Half: Normal lab
  • 2nd Half: Work independently using ChatGPT actively
    (this time we gave them tips!)

W08 lab

  • Normal lab (genAI tools are neither enforced nor barred)
  • Students complete a brief survey at the end of the session.

W09 lab

  • 1st Half: Normal lab
  • 2nd Half: Work independently using ChatGPT actively
    (this time we gave them tips!)

W10 lab

  • Normal lab (genAI tools are neither enforced nor barred)
  • Students complete a brief survey at the end of the session.

W11 lab

  • Normal lab (genAI tools are neither enforced nor barred)
  • Students complete a brief survey at the end of the session.

We set time off for ChatGPT usage

Sample testimonies from students

From the same lab (DS202A - W03):

Positive Experience

“I actually really enjoyed working with ChatGPT, as it helped me get past any chokepoints while still learning the syntax of the code and also exploring adjacent possibles in data science. I think it took my learning beyond the lesson plan and gave me ideas on how to use this in a more applied manner than just completing the tasks would have.” (DS202A W03)

Negative Experience

“ChatGPT was not very useful for me. I could answer the questions mostly by myself, with it maybe being useful for debugging, but I wasn’t able to understand what questions to ask so I got stuck :(” (DS202A W03)

Did they find ChatGPT useful?

How did they use ChatGPT?

How did they use ChatGPT? (Cont.)

Our key findings so far (qualitative):

  1. More context leads to better outcomes.
  2. The most relevant thing to emerge from the qualitative readings of the chat histories is not length but that they specify which packages they need to use.
  3. Students who find the tool more useful do not necessarily find they learn more. This seems to be related to a sentiment that those who successfully prompt the tool can rely too much on the code the tool generates. However: this is not the case for all students. (needs further investigation)

What’s Next?

Thank you!

The GENIAL team:

Photo of Dr Marcos Barreto
Dr. Marcos Barreto
Assistant Professor of Data Science (Education)
LSE Statistics
project lead
Photo of Dr Jon Cardoso Silva
Dr. Jon Cardoso-Silva
Assistant Professor of Data Science (Education)
LSE Data Science Institute
project lead
Photo of Dr Ghita Berrada
Dr. Ghita Berrada
Assistant Professor of Data Science (Education)
LSE Data Science Institute
core team
Photo of Dr Francesca Panero
Dr. Francesca Panero
Assistant Professor
LSE Statistics
core team
Photo of Ananya Reddi
Ananya Reddi
MSc Data Science (2022/23), LSE
Research Assistant
core team
Photo of Leonard Hinckeldey
Leonard Hinckeldey
MSc Applied Social Data Science (2022/23), LSE
Research Assistant
core team

The GENIAL project