By Niek de Schipper Research Engineer at the University of Amsterdam.
The recent accessibility to Generative AI in the form of chat bots has been rapidly transforming how people interact with information and creativity. Among its many applications, it has also gained significant attention in educational spaces, particularly for students at universities. However, despite its growing popularity, not much is known about how students are actually using these AI tools, especially in academic contexts. Are they relying on AI to generate ideas, conduct research, or simply as a tool for drafting their essays? And how do students themselves see that these tools are affecting their learning experience?
So far, action has been guided by anecdotal evidence about how students are using generative AI tools like ChatGPT to support their studies. An earlier survey at Utrecht University has suggested that it is rather widespread and identified several challenges with current policies. Formal research into how students are actually incorporating generative AI tools like ChatGPT into their academic workflows is still lacking. This is needed to create better policies and guidelines and to support the AI literacy needs of students and staff.
Research Questions
To bridge this knowledge gap, with data donations from ChatGPT, dr. Karin van Es and dr. Dennis Nguyen (from the Media and Culture Studies department/Utrecht University) aim to:
- Gain empirical insights into concrete uses of ChatGPT and how students interact with ChatGTP;
- explore how students perceive GenAI services such as ChatGPT in terms of their benefits, limitations, and risks
- provide universities and teaching staff with valuable information that allows them to address the role of GenAI in education.
This research offers valuable insights into how AI is shaping the future of education and helps educators make informed decisions about how to integrate or regulate AI usage in their courses.
To answer these questions, Karin, Dennis and their research assistant Ani Encheva collected and analyzed “conversations” that 15 students from the Humanities had with ChatGPT. Prior to the donation, the participants also complete a short survey which will help to contextualize the findings. To collect the data donation packages from participants, they used data donation with Port.
The Role of Data Donation in Answering These Questions
A promising method to explore these questions is through data donation, which enables researchers to gather real-world data on how participants are using AI tools like ChatGPT.
Under the General Data Protection Regulation (GDPR), individuals have the right to request their personal data from platforms that collect such information. In this study, students will first request and download their personal ChatGPT data from OpenAI. ChatGPT, like many large corporate platforms, have integrated easy buttons that enable this request in compliance with the regulation. Once obtained via email – in the case of ChatGPT, usually within minutes – they can share it with researchers. This dataset includes all conversations students had with ChatGPT, providing valuable insights into how they used and interacted with the tool.
Aside from challenges such as the willingness to donate and obtaining representative samples, several technical challenges accompany the process as well:
- The data can contain sensitive personal information, such as email addresses and phone numbers, raising privacy concerns.
- Additionally, the data is often presented in a JSON format, which can be difficult to interpret without appropriate tools or expertise.
Addressing these challenges is essential for ensuring the ethical use of data while gaining meaningful insights into student interactions with generative AI.
Data donation with Port
To address these questions, we have developed Port, a web application that operates entirely within the participant’s browser. Participants are invited to submit their downloaded data, and once submitted, the application extracts only the relevant information that aligns with the researcher’s interests. This extracted data is then presented on the screen for the participant to review. After reviewing the information, the participant can click on a “Yes, share for research” button. Only after this confirmation will the researcher receive the data that the participant has just reviewed.
So in the case of ChatGPT data this means that, before donation:
- Only the relevant information is extracted, specifically the conversations with ChatGPT.
- These conversations are displayed in a well-organized and searchable table, allowing students to review their conversations before deciding to share their data. This ensures that their consent is fully informed (The alternative would have been to allow students to share raw JSON files, which they might not fully understand.).
If you want to learn more about data donation check out datadonation.eu.
Want to see for yourself what students experience?
You can check out part of this study for yourself at the following link.
To try it out for yourself:
- Click “Start”
- Select “ChatGPT”
- Read the instructions on how to request and download your data
- Begin exploring your ChatGPT data (Note: this all happens in the browser, and no donations will take place)
I highly recommend giving it a try because it’s fun and low effort, as requesting and downloading your data from OpenAI is instantaneous.
Methodological Design: Gathering Data from Students
To facilitate the data donation process, the study was conducted (beginning October 2024) on-site with the RA who guided participants through downloading and donating their ChatGPT data. The RA explained the project and helped students address any technical difficulties that may arise during the download process or answer any questions they have. The presence of the RA is crucial, as it serves as a motivating factor for participants to complete the process, which can be demanding for some. Students who completed the survey and donated their data received a 15 euro Yetsy gift card in compensation for their time (the process takes between 30 minutes to an hour.)
Obtaining consent from students is essential to ensure they are fully informed about how their data will be used. In this study, special attention was given to the consent process due to the potential for the collected data to reveal instances of academic fraud. To safeguard participants, additional measures have been implemented: Utrecht University has signed a document assuring that the researchers will never be compelled to hand over the data for fraud or plagiarism.
Conclusion
In summary, this blog post emphasizes the potential impact of generative AI, especially ChatGPT, on students’ academic experiences while also highlighting the necessity for research (In this case to be executed by Karin and Dennis). The innovative data donation method enabled by Port provides a valuable way to gather insights while prioritizing participant privacy and informed consent.
If you have any questions or comments, please feel free to reach out!
Photo by Jonathan Kemper on Unsplash