GENIAL: GENerative AI Tools as a Catalyst for Learning
A Collaborative Focus Group

Note

The following pages have been recently updated (as of 12 January 2024):

• 📝 Methodology, describing the changes in our methodology for the Winter Term (Jan-March 2024).
• 🧾 Outputs with a slide we used at the LSE Data Science Institute Research Showcase.

Context

In higher education, students are increasingly using Generative AI tools like ChatGPT, GitHub Copilot, Bard, and Bing AI to aid their learning experience. These tools provide personalised, instant help with various tasks such as summarising literature, brainstorming ideas, and writing code and text, although some limitations in transparency and accuracy might exist. As educators, we recognise the potential of these tools and are considering how best to incorporate them into our teaching and assessments to support our students. However, we’re venturing into an area with limited concrete evidence regarding the effectiveness of these Generative AI tools in improving learning outcomes.

That’s the purpose of this focus group, led by Dr Marcos Barreto (LSE Department of Statistics) and Dr Jon Cardoso-Silva (LSE Data Science Institute). We want to explore the practical applications of these tools and understand how they might specifically enhance programming skills and critical thinking. We’re aiming to fill a knowledge gap and obtain insights that could provide valuable direction in how we approach education in light of these new technologies.

Objectives

This focus group draws on existing AI-based tools used for code generation and completion, including GitHub Copilot, and their use as a supporting programming tool in data science courses. The main goal is to prospectively assess the effectiveness of code-generation tools in undergraduate and postgraduate courses (see Case Studies).

Participants will engage in a literature review and group discussion to assess existing technologies. They will be expected to use their existing skills and knowledge to i) build meaningful and correct prompts, ii) critically assess the outputs against human-written code and literature examples, and iii) help in producing a set of curated examples, including prompts and curated outputs, that can be used to inform teaching practice and school policies on this matter.

We aim to address these initial questions:

• Surveying participants and the academic community for their experiences and expectations related to generative AI.
• Reviewing the literature on generative AI tools in Education.
• Identifying suitable tools for data science/quantitative courses.
• Testing selected tools against reference examples and establishing assessment metrics.
• Implementing and validating case studies.
• Producing evidence to support peers and inform policy decisions

Timeline

ONLINE ACTIVITIES

• June 2023: focus group proposal approved, call for participation launched.
• July 2023: students’ selection, literature search (papers and tools), project revision & planning.
• August 2023: technology tests based on existing examples (seminar sessions + literature), discussion of case studies.

IN-PERSON ACTIVITIES

• September 2023: experiment planning (ST207, DS105A, DS202A)
• October/December 2023:
  • deployment of case studies (ST207, DS105A)
  • evidence collection
  • experiment planning (ST456, DS105W, DS202W)
  • report draft.
• January/March 2024:
  • deployment of case studies (ST456, DS105W, DS202W)
  • evidence collection
  • report writing
  • draft of dissemination materials.
• April 2024: dissemination (blog, webinar, technical paper).