🗓️ 2023/24 Autumn Term – Activities
What happened in the 
GENIAL project between September and December 2023
How we ran the experiments
Starting Week 03 through Week 11, students completed a brief survey after each lab session. This survey collected their immediate insights on the session’s learning outcomes and overall course experience. Students also submitted their chat logs, screenshots, and lab work on days featuring an experiment.
We (tried to) run four observational experiments during Autumn term, spread across the three participating courses (ST207, DS105A, DS202A). These occured during small-group classes, which host 10-30 students and last 1.5 hours each week. Generally, students were asked to tackle exercises that built upon concepts covered in the preceding lecture.
Structure of the labs
Lab sessions in key weeks were split into two halves. In the first half, all students – participants or not – worked on the lab exercises as usual. In the second half, all students worked solo to solve a few more challenging problem sets, but unlike the rest of the class, participants were asked not to use the Web freely. They could not use search engines or seek assistance on forums or Q&A sites (such as StackOverflow, commonly used for programming help). Our goal was to isolate the experience of engaging with a generative AI tool as an ‘assistant’ to the lab exercises and to capture participant’s thoughts on its usefulness and efficacy for learning. At the end of the session, participants filled out a brief survey on and submitted their chat logs.
Especially for the first two experiments (around W03 & W04), class teachers were instructed to provide tips and assistance as usual on the course content and lab exercises but not to give information on ChatGPT or prompt engineering. Our main goal for the first few weeks was to observe how students use the tool, whether they are already proficient with it or not.
The second half of the Term was different
The dynamic is different in the final two observational experiments, planned to take place in W07 and W09 of the academic Term.
On Week 061, we sent a newsletter to all participants with text and links about:
- the common criticisms of generative AI tools in education
- ‘prompt engineering’ tricks to get the most out of generative AI tools
These contrasting sets of resources delineate a new phase of the study. We will now observe how students use generative AI tools when they are hopefully made more aware of the criticisms and ways of getting the most out of them. How does the perception of the usefulness of these tools as an in-class learning aid change?
After this, while the structure of labs will remain the same, class teachers and lecturers can give information on prompt engineering if students ask for it.
Initially, we wanted to run the first two labs as described above and then run Randomised Controlled-Trials on the remaining two. We would separate participants into those allowed to use generative AI tools versus those permitted to use generative AI tools versus those who cannot. This way, it would have been easier for their learning outcomes and experiences.
However, although we were happy to have ~55 sign-ups for the study, our participants were spread across three courses (see this plot) which made it difficult to run the RCTs. At each seminar/lab, we had only a handful of participants who signed up for the study, which meant that we could not split them into two groups and still have a meaningful sample size.
Schedule
Participating Courses
Timeline
On top of the initial opt-in survey and the final, end-of-term survey, we asked students to reflect on their learning in the classroom with hands-on programming exercises that are typical of the participating courses. Below was our tentative timeline of the experiments we ran this term. In practice, we didn’t manage to run all of them (more on that in the future).
W01 lab
Normal lab (no enforcement of ChatGPT)
- We will be promoting the 📋 Opt-In Survey
W02 lab
Normal lab (no enforcement of ChatGPT)
- We will be promoting the 📋 Opt-In Survey
W03 lab
After introducing a new concept/package/piece of software, we are ready to run the first experiment:
- 1st Half: Normal lab
- 2nd Half: Work independently
(participants will be asked to work alone, but actively using ChatGPT however they like.)
Participants will later export their chat logs, send them to us, and complete a short survey.
W04 lab
After introducing a new concept/package/piece of software, we are ready to run the first experiment:
- 1st Half: Normal lab
- 2nd Half: Work independently
(participants will be asked to work alone, but actively using ChatGPT however they like.)
Participants will later export their chat logs, send them to us, and complete a short survey.
W05 lab
Normal lab (generative AI tools are neither enforced nor barred)
- Students will still complete the same brief survey at the end of the lab session.
W07 lab
After introducing a new concept/package/piece of software, we are ready to run the first experiment:
- 1st Half: Normal lab
- 2nd Half: Work independently
(participants will be asked to work alone, but actively using ChatGPT (or, depending on the course, GitHub Copilot) however they like.)
Participants will later export their chat logs, send them to us, and complete a short survey.
W08 lab
Normal lab (generative AI tools are neither enforced nor barred)
- Students will still complete the same brief survey at the end of the lab session.
W09 lab
After introducing a new concept/package/piece of software, we are ready to run the first experiment:
- 1st Half: Normal lab
- 2nd Half: Work independently
(participants will be asked to work alone, but actively using ChatGPT (or, depending on the course, GitHub Copilot) however they like.)
Participants will later export their chat logs, send them to us, and complete a short survey.
W10 lab
Normal lab (generative AI tools are neither enforced nor barred)
- Students will still complete the same brief survey at the end of the lab session.
W11 lab
Normal lab (generative AI tools are neither enforced nor barred)
- Students will still complete the same brief survey at the end of the lab session.
Footnotes
Week 06 is Reading Week in all participating courses. Students do not have lectures or classes and are either taking some time off or working on course assignments.↩︎