šŸ“‹ Initial Survey

Last Updated: 25 September 2023

The following is a copy of the MS Office Forms that will serve as our initial questionnaire for students who want to take part in this course.

GENIAL: Participant Opt-in Survey

šŸ“£ You donā€™t need to have any prior experience with programming or generative AI tools like ChatGPT to participate in this study. We are interested in understanding precisely if, when and how these tools can be beneficial in learning contexts. We welcome participants of all backgrounds to join our research. You can read more about the purposes of this research at https://lse-dsi.github.io/genial.

Why participate?

  • Your experience in this study might shape future policy around the use of generative AI in the classroom at LSE and beyond.
  • You will be contributing to academic research.
  • You will get a chance to experiment with cutting-edge technology.

Terms & Conditions

šŸ‘‰ By filling out this form, you agree to participate in this study and with the terms and conditions listed below.

We know this is long! However, it is important that you read and understand all the terms of participating in this study.

  1. You agree to fill out two comprehensive surveys: this initial survey and a final survey at the end of the Autumn Term.

  2. You agree to fill out our brief surveys (3-5 min) weekly, at the end of each participating class. These surveys will ask you about your experience in the class, even in days where we donā€™t have any generative AI experiment, and will only start in Week 03 of the Autumn Term.

  3. You agree to participate in the four experiments throughout your participating course (provisionally: Weeks 03, 04, 07 and 09 of the Autumn Term).

  4. The experiments will happen during regular small-group classes (also called labs, tutorials, seminars, etc.). Everyone in the classroom will be solving the same problem sets, but participating members of GENIAL will be given some requirements. For example, you might be asked to work independently and use ChatGPT to help you with the answers, or you might be specifically asked NOT to use ChatGPT (control group).

  5. On the days of the experiments, you might be asked to attach evidence of your use of a generative AI tool, such as exporting and uploading the log of a chat you created during a class, or screenshots of the use of another tool. You agree to give us permission to store and study these records.

  6. You agree to inform us of your LSE Candidate Number every time you fill out one of the surveys. This is so we can link your responses together, helping us identify any interesting patterns in the data. At the end of the academic year, we will further anonymize your records so they appear as random strings, such as s45a1a65s897s, making it nearly impossible to trace them back to you as an individual.

  7. Your data will be primarily studied in aggregate with the data of all participant members. We might draw statistical conclusions, and perform text mining or other suitable techniques to uncover patterns in the data.

  8. We might use the text or prints of your individual chat interactions, evidence of use of a specific tool and responses to a survey if relevant to illustrate a point in one of our reports, presentations, public speaking engagements or academic writing. We will take extra precautions to never allow any identifiable information (such as your LSE Candidate Number) to show up during any of these use cases.

  9. You can opt-out at any moment, no questions asked. To opt-out, send an e-mail to the class convenor to the participating course you are taking. If you opt out, we will aim to completely remove your data from our records within 14 days of receipt of your message.

Q1

Do you agree to take part in this study under the terms and conditions listed above?

General

We need a way to link all your responses during the study. We will use your student number during the academic year but then, we will further anonymise your records so that instead of e.g. 2023015748, your records will show up as something seemingly random like s45a1a65s897s, leaving no way to trace it back to you.

Q2

What is your student number?

This is a unique identifier given when joining LSE, starting with the year of enrollment and looking like 2023015748.

šŸ“£ An important note on anonymity: we need a way to link all your responses during the study. At the end of the academic year, however, we will further anonymize your records so they appear as random strings, such as s45a1a65s897s, making it impossible to trace them back to you.

(Free text)

Q3

What is your degree programme?

(Free text)

Q4

What Year of Study are you in?

  • I am on my 1st Year
  • I am on my 2nd Year
  • I am on my 3rd Year
  • I am on my 4th Year

Q5

Please indicate which of the participating courses below you are enrolled in.

(You can only take part in this study if you are enrolled in at least one of the participating courses listed.)

Your prior experience with programming

All courses involved in this study touch on programming skills to varying degrees. Whether itā€™s Python, R, or SQL, some courses start from scratch (for example, SQL in ST207), while others assume a basic understanding (such as DS202 with R basics).

Donā€™t worry, you donā€™t need prior programming skills to participate in this study. However, we are interested in gauging your current level of proficiency. This will help us assess if prior knowledge significantly influences learning outcomes when an AI tool is involved.

Q6

Irrespective of the programming language, did you know the basics of programming before enrolling in the participating course(s)?

By ā€˜basicsā€™ we mean: the ability to create variables, write basic for and while loops, and create custom functions (for example, the def operator in Python)

Branching: if the answer is ā€˜Noā€™, skip to Q6.

Q7

Which programming language(s) did you know (or are learning) before the start of the Autumn Term?

Q8

How do you feel, in general, about programming?

For example, is it something you enjoy, did you only pick up programming because you were told to learn, etc.?

(Free text)

Q9

What is the Operating System of the computer you will be using the most during this course?

Your Learning Process in the Classroom

Before delving into questions about generative AI tools, we would like to gain insights into your learning process. The questions below pertain to small group classes, including labs, seminars, tutorials, or workshops (not lectures).

For the questions in this section, think of your most recent experiences attending a small group class. Answer them as honestly as possible.

Q10

Thinking back to your most recent experiences attending small group classes, did you prepare beforehand by reviewing class materials or watching pre-recorded lectures, etc.?

Branching: if the answer is ā€˜Alwaysā€™, go to Q9. If the answer is ā€˜Sometimesā€™, go to Q10. Otherwise, go to Q11.

Q11

Why did you find it important to study class materials beforehand?

(Free text)

Branching: go to Q12.

Q12

What specifically motivated you to prepare for a class beforehand?

(Free text)

Branching: go to Q12.

Q13

Why didnā€™t you feel motivated to prepare for these classes in advance?

(Free text)

Branching: go to Q12.

Q14

Thinking back to your most recent experiences during small group classes, which of the following methods did you find to be more effective for your learning during the session?

Rank these options from most important (top) to least important (bottom).

  • Reading the class materials (workshop handouts, lab tutorials, etc.) alone
  • Discussing with other students
  • Attempting to solve problems (writing code, equations, text, etc.) on my own
  • Attempting to solve problems with other students
  • Actively listening to the class teacher

(The order of the options is shuffled for each participant.)

Q15

Were there any other methods, not mentioned in the previous question, during a small group class that you believe were even more effective for your learning?

(Free text)

Q16

Thinking back to your experiences during a small group class, when you felt the need to revisit a concept or gather more information about a topic, what were you most likely to do?

Rank these options from most likely (top) to least likely (bottom).

  • Visiting a specific website (e.g., Wikipedia, StackOverflow, etc.)
  • Using a search engine (e.g., Google, DuckDuckGo, Bing, etc.)
  • Checking/searching social media feeds (e.g., Twitter, LinkedIn, Instagram, Weibo, TikTok, Facebook, etc.)
  • Using an AI chatbot (e.g., ChatGPT, etc.)
  • Asking questions to the class teacher
  • Asking questions to other students
  • Consulting the class materials (workshop handouts, lab tutorials, etc.)
  • Consulting academic materials (e.g., articles, textbooks, books, etc.)
  • Consulting the course Moodle page

(The order of the options is shuffled for each participant.)

Q17

Here we want to know how you would assess whether you learned something new after attending a class.

Rank the items below from most important (top) to least important (bottom) factors that best helped you determine that youā€™ve truly learned something new in a class:

  • When I can recall information from memory
  • When I can apply what Iā€™ve learned to solve various problems.
  • When I can reproduce what Iā€™ve learned (e.g., write code, solve an equation) without needing reference material.
  • When I can explain what Iā€™ve learned to others
  • Based on my emotional response to the class topic (more of a feeling than a thought)

(The order of the options is shuffled for each participant.)

Q18

Were there any other factors apart from those mentioned in the previous question?

(Free text)

Q19

Thinking of your most recent experiences attending small group classes, did you typically allocate time to review class materials after the session?

Branching: if the answer is ā€˜Alwaysā€™, go to Q20. If the answer is ā€˜Sometimesā€™, go to Q21. Otherwise, go to Q22.

Q20

Why did you find it important to review class materials after attending the class?

(Free text)

Branching: go to Q23.

Q21

What triggered this need to review the class materials?

(Free text)

Branching: go to Q23.

Q22

Was it true even when you struggled with the class topic?

(Free text)

Branching: go to Q23.

Generative AI Tools

Now we want to know a bit more about your use and knowledge of generative AI tools.

Generative AI refers to software capable of producing synthetic content, such as text, images, audio, or video. Machine learning algorithms power these tools, which are trained on vast datasets comprising human-created content typically scraped from the web. For example, ChatGPT generates text, having been trained on a large corpus of online data and further refined through user interaction. Similarly, DALLĀ·E produces images, having been trained on a massive dataset of human-generated imagery.

Q23

Below is a list of current notable generative AI tools. Which of these tools have you heard about?

Check all that apply.

(The order of the options is NOT shuffled)

Branching is unavailable on multi-choice questions, so I added a ā€˜Not applicableā€™ option in the next question.

Q24

Which of the following generative AI tools have you experimented with before?

Check all that apply.

(The order of the options is NOT shuffled)

Branching is unavailable on multi-choice questions, so I added a ā€˜Not applicableā€™ option in the next question as well.

Q25

Have you ever tried to use a generative AI tool to help you learn or master a topic in a learning environment before (reviewing school or university subjects, studying for an assessment, etc.)?

(Weā€™re not asking if you were successful, just if you tried.)

Branching: if the answer is ā€˜Yesā€™, go to Q26. Otherwise, go to Q29.

Q26

Thinking of this experience, do you feel the generative AI tool made it easier or more difficult to learn the topic, as opposed to not using any generative AI tool?

Q27

Do you think introducing generative AI tools in a small group class can help improve the learning experience?

Q28

Thinking of the following tasks that are normally part of a small-group class, which ones do you think a generative AI tool could help you with the most?

Rank the items below from most relevant (top) to least relevant (bottom).

  • Reading and summarising the class materials (lab tutorials, walkthroughs, seminar handouts, etc.)
  • Clarifying the content of class materials (lab tutorials, walkthroughs, seminar handouts, etc.)
  • Clarifying questions in problem sets or lab tutorials
  • Solving class exercises
  • Helping you think about how to solve the problem (without solving it for you)

The End

Q29

Is there anything that we didnā€™t ask but you would like to tell us about? Or any suggestions about this study?