The latest development of Generative Artificial Intelligence (GenAI), particularly ChatGPT, has drawn the attention of educational researchers and practitioners. We have witnessed many innovative uses of ChatGPT in STEM classrooms. However, studies regarding students’ perceptions of ChatGPT as a virtual tutoring tool in STEM education are rare. The current study investigated undergraduate students’ perceptions of using ChatGPT in a physics class as an assistant tool for addressing physics questions. Specifically, the study examined the accuracy of ChatGPT in answering physics questions, the relationship between students’ ChatGPT trust levels and answer accuracy, and the influence of trust on students’ perceptions of ChatGPT. Our finding indicates that despite the inaccuracy of GenAI in question answering, most students trust its ability to provide correct answers. Trust in GenAI is also associated with students’ perceptions of GenAI. In addition, this study sheds light on students’ misconceptions toward GenAI and provides suggestions for future considerations in AI literacy teaching and research.

With the exponential development and vast interest in artificial intelligence (AI), the global economic impact of AI is expected to reach $15.7 trillion by 2030. While AI has infiltrated everyday life, a lack of knowledge of what AI is and how AI works is ubiquitous across all ages and professions. Teaching AI literacy to non-technical individuals has become imperative and requires immediate attention, however, assessing AI literacy has heavily relied on subjective measurements such as qualitative assessment and self-reported surveys, which may lead to biased results. This study contributes to the field by developing and validating an assessment created based on a well-established AI literacy framework. A total of 196 responses were collected from pre-and in-service teachers in the United States, and 186 responses were included in the analysis to validate the assessment. The final assessment comprises 25 objective-based items reduced from an originally 31-item assessment. Both experts’ insights were sought, and statistical methodology was employed to ensure the validity of the assessment. The results indicate that pre-and in-service teachers have a moderate level of AI literacy and in-service teachers performed slightly better than pre-service teachers on our assessment. Inconsistent answers across various AI concepts indicate that teachers may possess an even more ambiguous understanding of certain AI concepts.