Methodology
Last updated: 2 November 2023
How we run the experiments
Starting Week 03 through Week 11, students will complete a brief survey after each lab session. This survey will gauge their immediate insights on the session’s learning outcomes and overall course experience. Students will also submit their chat logs, screenshots, and lab work on days featuring an experiment.
We are running four observational experiments this Autumn term, spread across our three courses. These will occur during small-group classes, which host 10-30 students and last 1.5 hours each week. Generally, students will tackle exercises that build upon concepts covered in the preceding lecture.
Structure of the labs
Each lab session will be split into two halves. In the first half, all students – participants or not – will work on the lab exercises as usual. In the second half, all students will work solo to solve a few more challenging problem sets, but unlike the rest of the class, participants will be asked not to use the Web freely. They cannot use search engines or seek assistance on forums or Q&A sites (such as StackOverflow, commonly used for programming help). With this, we aim to isolate the experience of engaging with a generative AI tool as an ‘assistant’ to the lab exercises and to capture participant’s thoughts on its usefulness and efficacy for learning. At the end of the session, participants fill out a brief survey on and submit their chat logs.
Especially for the first two experiments (around W03 & W04), class teachers were instructed to provide tips and assistance as usual on the course content and lab exercises but not to give information on ChatGPT or prompt engineering. Our main goal for the first few weeks is to observe how students use the tool, whether they are already proficient with it or not.
The second half of the Term is different
The dynamic is different in the final two observational experiments, planned to take place in W07 and W09 of the academic Term.
On Week 061, we sent a newsletter to all participants with text and links about:
- the common criticisms of generative AI tools in education
- ‘prompt engineering’ tricks to get the most out of generative AI tools
These contrasting sets of resources delineate a new phase of the study. We will now observe how students use generative AI tools when they are hopefully made more aware of the criticisms and ways of getting the most out of them. How does the perception of the usefulness of these tools as an in-class learning aid change?
After this, while the structure of labs will remain the same, class teachers and lecturers can give information on prompt engineering if students ask for it.
Initially, we wanted to run the first two labs as described above and then run Randomised Controlled-Trials on the remaining two. We would separate participants into those allowed to use generative AI tools versus those permitted to use generative AI tools versus those who cannot. This way, it would have been easier for their learning outcomes and experiences.
However, although we were happy to have ~55 sign-ups for the study, our participants were spread across three courses (see this plot) which made it difficult to run the RCTs. At each seminar/lab, we had only a handful of participants who signed up for the study, which meant that we could not split them into two groups and still have a meaningful sample size.
Schedule
1️⃣ Autumn Term 2023/24
Participating Courses
Timeline
On top of the initial opt-in survey and the final, end-of-term survey, we will ask students to reflect on their learning in the classroom with hands-on programming exercises that are typical of the participating courses. Below is a tentative timeline of the experiments we will run this term.
W01 lab
Normal lab (no enforcement of ChatGPT)
- We will be promoting the 📋 Opt-In Survey
W02 lab
Normal lab (no enforcement of ChatGPT)
- We will be promoting the 📋 Opt-In Survey
W03 lab
After introducing a new concept/package/piece of software, we are ready to run the first experiment:
- 1st Half: Normal lab
- 2nd Half: Work independently
(participants will be asked to work alone, but actively using ChatGPT however they like.)
Participants will later export their chat logs, send them to us, and complete a short survey.
W04 lab
After introducing a new concept/package/piece of software, we are ready to run the first experiment:
- 1st Half: Normal lab
- 2nd Half: Work independently
(participants will be asked to work alone, but actively using ChatGPT however they like.)
Participants will later export their chat logs, send them to us, and complete a short survey.
W05 lab
Normal lab (generative AI tools are neither enforced nor barred)
- Students will still complete the same brief survey at the end of the lab session.
W07 lab
After introducing a new concept/package/piece of software, we are ready to run the first experiment:
- 1st Half: Normal lab
- 2nd Half: Work independently
(participants will be asked to work alone, but actively using ChatGPT (or, depending on the course, GitHub Copilot) however they like.)
Participants will later export their chat logs, send them to us, and complete a short survey.
W08 lab
Normal lab (generative AI tools are neither enforced nor barred)
- Students will still complete the same brief survey at the end of the lab session.
W09 lab
After introducing a new concept/package/piece of software, we are ready to run the first experiment:
- 1st Half: Normal lab
- 2nd Half: Work independently
(participants will be asked to work alone, but actively using ChatGPT (or, depending on the course, GitHub Copilot) however they like.)
Participants will later export their chat logs, send them to us, and complete a short survey.
W10 lab
Normal lab (generative AI tools are neither enforced nor barred)
- Students will still complete the same brief survey at the end of the lab session.
W11 lab
Normal lab (generative AI tools are neither enforced nor barred)
- Students will still complete the same brief survey at the end of the lab session.
2️⃣ Winter Term
Unless we spot any blind spots in the Autumn term, we will run the experiments similarly here.
This website section will be updated closer to the start of the Winter Term.
Footnotes
Week 06 is Reading Week in all participating courses. Students do not have lectures or classes and are either taking some time off or working on course assignments.↩︎