Enhancing Chemistry Learning with ChatGPT, Bing Chat, Bard, and Claude as Agents-to-Think-With

Table of Links

Abstract and Introduction

Materials And Methods

Results and Analyses

Prompts and generated texts

Conceptualizing chemical reactions

Deepening on understanding of chemical reactions

Question about combustion

Question about a graph of gases turning into water over time

Question about the difference between atoms, molecules, and moles

Deepening on the concept of mole

Question about changing of state

Question about an animated representation of water molecules undergoing phase changes

Question about plasma, a state of matter

Question about chemical bondings

Question about illustration of chemical bonds

Question about the essence of the type of chemical bonding

Further analysis

Conclusions

Limitations of the study and possible future studies

Author Contributions, Conflicts of interest, Acknowledgements, and References

Abstract

This research delves into the comparative advantages of Generative AI chatbots (GenAIbots) - ChatGPT, Bing Chat, Bard, and Claude - in the context of Chemistry education, framed within a constructivist perspective. Our primary objective was to identify which of these four AI tools is more effective for enhancing Chemistry learning. Employing a single-case study approach, we scrutinised interaction logs between the AI systems and a simulated student persona during Chemistry learning simulations, incorporating Content Analysis methodology to delve deeper into the discourse. Our findings underscore these tools' potential as "agents-to-think-with", enhancing critical thinking, problem-solving, comprehension, creativity, and tailored learning. Especially noteworthy is their ability to stimulate learners through Socratic-like questioning, aligning with constructionist principles. The research emphasises the pivotal role of prompt crafting to coax desired responses from GenAIbots, engendering iterative reflections. It also highlights the need for robust educator training to infuse these technologies into educational settings. Conclusively, while ChatGPT, Bing Chat, Bard, and Claude are poised to enrich Chemistry education by fostering dynamic, inclusive learning experiences, ChatGPT stood out, decisively surpassing Bing Chat's performance. Bard and Claude trailed closely, with all three showcasing a more in-depth, precise, and nuanced understanding, underscoring ChatGPT's adeptness at contextual comprehension.

Keywords: Chemistry education, ChatGPT, Bing Chat, Bard, Claude, Artificial Intelligence in Education, agents-to-think-with

Introduction

Chemistry, a core Science Education subject, clarifies matter's properties and transformations, thus critically shaping our daily lives (Dunlop et al., 2020). However, Chemistry is also a complex and dynamic field that necessitates a profound grasp of fundamental concepts and principles, and it is sometimes difficult for students to associate real-life circumstances with abstract chemistry concepts (Dewi et al., 2021). Research in Chemistry Education seeks effective strategies to alleviate these learning difficulties (Permatasari et al., 2022; Timilsena et al., 2022; Tümay, 2016).

Timilsena et al. (2022) identified difficulties in understanding the abstract nature of chemical reactions and factors such as inadequate teaching materials and curriculum complexity, emphasising the need for effective teaching strategies and tools. Tümay (2016) discussed students' struggles in understanding fundamental chemistry concepts and emphasised the importance of addressing misconceptions and learning difficulties.

Dewi et al. (2021) underscored the need for critical thinking skills and the integration of digital technology to enhance the quality of Chemistry education for Generation Z students. Dunlop et al. (2020) proposed introducing philosophical dialogue in higher education to address undergraduate Chemistry students' challenges, suggesting that it can stimulate new ways of thinking.

In 2023, Castro Nascimento and Pimentel undertook a study to evaluate the proficiency of the ChatGPT model by having it respond to five distinct tasks across various subfields of chemistry (Castro Nascimento & Pimentel, 2023). These tasks included converting compound names to their SMILES chemical representation and vice versa, procuring information on the octanol-water partition coefficient of chemical compounds, extracting structural information on coordination compounds, determining the water solubility of polymers, and identifying the molecular point groups of simple molecular compounds. The unsatisfactory outcomes highlighted potential limitations in the model's ability to adequately address these specific chemistry-related queries. Notably, the suboptimal performance observed might be attributed to utilising an antiquated version of ChatGPT, specifically the GPT-3 model introduced by OpenAI in 2020 (Brown et al., 2020).

In a separate study conducted in the same year, Leon and Vidhani (2023) explored the reliability of ChatGPT responses within the framework of an introductory college-level Chemistry course. Their findings indicated a significant reliability concern, with ChatGPT failing to secure a score above 37%. Such a performance implies that learners relying on this tool for study support would predominantly receive incorrect responses, with the variability of the tool leading to different responses for individual learners. Although the authors didn't specify which version of ChatGPT they utilised in their study, their reference to "ChatGPT's free original version" and the citation of Floridi and Chiriatti's (2020) research on ChatGPT-3 hint at the likelihood that they employed this version.

Additionally, Pimentel et al. (2023) delved into the efficacy of ChatGPT versions 3 and 4 in responding to intricate questions spanning six topics in Chemistry. While they concluded that both versions were currently inadequate in addressing the nuances of complex topics, they also observed noteworthy advancements from ChatGPT-3 to ChatGPT-4. Such progress signals promising potential for the tool to aid scientists in future literature reviews, surveys, and educational endeavours.

Generative AI-powered chatbots (GenAIbots), including ChatGPT, Bing Chat, Bard, and Claude, have been introduced as innovative solutions to persistent challenges in Chemistry education (Baidoo-Anu & Owusu Ansah, 2023; Taylor et al., 2022). Their emergence offers a dynamic, inclusive educational environment, transforming how complex concepts are conveyed and understood. These GenAIbots simplify intricate topics, promote self-reflection, engage users in stimulating dialogues, facilitate personalised learning, and enhance critical thinking, collaboration, and cognitive development (Okonkwo & Ade-Ibijola, 2021). As a result, they have played a pivotal role in revolutionising Chemistry education.

In the discourse on agency, traditional theories, such as the ones of Anscombe and Davidson, attribute actions to entities based on representational mental states encompassing desires, beliefs, and intentions. However, alternative perspectives, drawing from the works of Heider and Simmel, Dennett, Davidson, and Barandiaran et al., challenge this representation dependent view, suggesting potential agency without such mental representations (Schlosser, 2019). When considering GenAIbots within these frameworks, the entity's ability to respond based on training contrasts with its lack of self-awareness, intentions, and proactive behaviour— characteristics central to the agency. Consequently, while dominant philosophical and cognitive models would not grant ChatGPT agency, more expansive interpretations, as offered by these cited thinkers, could accept it.

Building on this foundation and inspired by Melanie Swan's suggestion, we have expanded upon Papert's (1980) concept of "objects-to-think-with" to introduce “agents-to-thinkwith.” This positions GenAIbots as integral participants in the educational path, echoing Turkle's (1984) concept of "metacognitive machines," in the sense that they assist in enhancing one's awareness of their own cognitive processes, i.e., "thinking about thinking" as described by Flavell (1976). In Latour's (1991) perspective, we can perceive GenAIbots as "hybrids," which blur the boundaries between humans and non-human entities. Such a perspective further aligns with Swan's (2015) vision of a future where augmented humans and AI collaborate and thrive symbiotically.

While GenAIbots posit themselves as powerful metacognitive educational tools, promoting critical thinking, problem-solving, and deep comprehension of concepts, it's imperative to acknowledge their limitations. For instance, there's the possibility of generating nonsensical or inaccurate content, as OpenAI (2023) highlighted. Nevertheless, the unique ability of GenAIbots to provide instant feedback, introduce diverse perspectives, and foster an interactive engagement with complex ideas solidifies their position as influential agents-to-think-with in the realm of Chemistry education.