Joanne Kim

Joanne Kim

Auto-generated profile, based on publicly available information

Joanne Kim

Temple University, Philadelphia, PA, USA

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
29 June 2023
Verified
Conference Paper
Open access
JLJuho Leinonen
Juho Leinonen
PDPaul Denny
Paul Denny
Stephen Michael MacNeilStephen Michael MacNeil
Stephen Michael MacNeil
SSSami Sarsa
Sami Sarsa
SBSeth Bernstein
Seth Bernstein
AHArto Hellas
Arto Hellas
Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course (n ≈ 1000) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.

2 March 2023
Verified
Conference Paper
Open access
Stephen Michael MacNeilStephen Michael MacNeil
Stephen Michael MacNeil
ATAndrew Tran
Andrew Tran
AHArto Hellas
Arto Hellas
JKJoanne Kim
Joanne Kim
SSSami Sarsa
Sami Sarsa
JLJuho Leinonen
Juho Leinonen
Advances in natural language processing have resulted in large language models (LLMs) that can generate code and code explanations. In this paper, we report on our experiences generating multiple code explanation types using LLMs and integrating them into an interactive e-book on web software development. Three different types of explanations -- a line-by-line explanation, a list of important concepts, and a high-level summary of the code -- were created. Students could view explanations by clicking a button next to code snippets, which showed the explanation and asked about its utility. Our results show that all explanation types were viewed by students and that the majority of students perceived the code explanations as helpful to them. However, student engagement varied by code snippet complexity, explanation type, and code snippet length. Drawing on our experiences, we discuss future directions for integrating explanations generated by LLMs into CS classrooms.

6 March 2022
Verified
Conference Paper
Open access
Stephen Michael MacNeilStephen Michael MacNeil
Stephen Michael MacNeil
ATAndrew Tran
Andrew Tran
JLJuho Leinonen
Juho Leinonen
PDPaul Denny
Paul Denny
JKJoanne Kim
Joanne Kim
SSSami Sarsa
Sami Sarsa
Recent breakthroughs in Large Language Models (LLMs), such as GPT-3 and Codex, now enable software developers to generate code based on a natural language prompt. Within computer science education, researchers are exploring the potential for LLMs to generate code explanations and programming assignments using carefully crafted prompts. These advances may enable students to interact with code in new ways while helping instructors scale their learning materials. However, LLMs also introduce new implications for academic integrity, curriculum design, and software engineering careers. This workshop will demonstrate the capabilities of LLMs to help attendees evaluate whether and how LLMs might be integrated into their pedagogy and research. We will also engage attendees in brainstorming to consider how LLMs will impact our field.

29 January 2024
Verified
Conference Paper
Open access
Stephen Michael MacNeilStephen Michael MacNeil
Stephen Michael MacNeil
PDPaul Denny
Paul Denny
ATAndrew Tran
Andrew Tran
JLJuho Leinonen
Juho Leinonen
SBSeth Bernstein
Seth Bernstein
JKJoanne Kim
Joanne Kim
Identifying and resolving logic errors can be one of the most frustrating challenges for novices programmers. Unlike syntax errors, for which a compiler or interpreter can issue a message, logic errors can be subtle. In certain conditions, buggy code may even exhibit correct behavior – in other cases, the issue might be about how a problem statement has been interpreted. Such errors can be hard to spot when reading the code, and they can also at times be missed by automated tests. There is great educational potential in automatically detecting logic errors, especially when paired with suitable feedback for novices. Large language models (LLMs) have recently demonstrated surprising performance for a range of computing tasks, including generating and explaining code. These capabilities are closely linked to code syntax, which aligns with the next token prediction behavior of LLMs. On the other hand, logic errors relate to the runtime performance of code and thus may not be as well suited to analysis by LLMs. To explore this, we investigate the performance of two popular LLMs, GPT-3 and GPT-4, for detecting and providing a novice-friendly explanation of logic errors. We compare LLM performance with a large cohort of introductory computing students (n = 964) solving the same error detection task. Through a mixed-methods analysis of student and model responses, we observe significant improvement in logic error identification between the previous and current generation of LLMs, and find that both LLM generations significantly outperform students. We outline how such models could be integrated into computing education tools, and discuss their potential for supporting students when learning programming.

6 March 2022
Verified
Conference Paper
Open access
Stephen Michael MacNeilStephen Michael MacNeil
Stephen Michael MacNeil
JKJoanne Kim
Joanne Kim
JLJuho Leinonen
Juho Leinonen
PDPaul Denny
Paul Denny
SBSeth Bernstein
Seth Bernstein
VKViraj Kumar
Viraj Kumar
The introduction of Large Language Models (LLMs) has generated a significant amount of excitement both in industry and among researchers. Recently, tools that leverage LLMs have made their way into the classroom where they help students generate code and help instructors generate learning materials. There are likely many more uses of these tools -- both beneficial to learning and possibly detrimental to learning. To help ensure that these tools are used to enhance learning, educators need to not only be familiar with these tools, but with their use and potential misuse. The goal of this BoF is to raise awareness about LLMs and to build a learning community around their use in computing education. Aligned with this goal of building an inclusive learning community, our BoF is led by globally distributed discussion leaders, including undergraduate researchers, to facilitate multiple coordinated discussions that can lead to a broader conversation about the role of LLMs in CS education.