Ashley Pang

Ashley Pang

Auto-generated profile, based on publicly available information

Ashley Pang

University of California, Riverside, Riverside, CA, USA

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
3 July 2024
Verified
Conference Paper
Open access
APAshley Pang
Ashley Pang
FVFrank Vahid
Frank Vahid
We experimented with ChatGPT's ability to write programs in a CS1 class, and the ability of a popular tool to auto-detect ChatGPT-written programs. We found ChatGPT was proficient at generating correct programs from a mere copy-paste of the English programming assignment specifications. However, running ChatGPT for 10 programming assignments and acting as 20 different students, and using zyBook's APEX beta tool for academic integrity, we found: (1) ChatGPT-generated programs tend to use a programming style departing from the style taught in the textbook or by the instructor, and these "style anomalies" were automatically detected. (2) Although ChatGPT may for the same assignment generate a few different program solutions for different students, ChatGPT often generates highly-similar programs for different students, so if enough students in a class (e.g., 5 or more) use ChatGPT, their programs will likely be flagged by a similarity checker. (3) If students are required to do all programming in the autograder's IDE, then a student using ChatGPT ends up showing very little time relative to classmates, which is automatically flagged. (4) Manually, we observed that if a student consistently uses ChatGPT to submit programs, the programming style may vary across programs, something normal students don't do; automation of style inconsistency detection was recently added to APEX. In short, while there will no doubt be an arms race between AI-generated programs and automatic detection of AI-generated programs, currently students using ChatGPT for multiple CS1 programs can be detected by automated tools such as zyBooks' APEX.

7 March 2024
Verified
Conference Paper
Open access
FVFrank Vahid
Frank Vahid
APAshley Pang
Ashley Pang
BDBenjamin Denzler
Benjamin Denzler
Automated assistance for detecting cheating on programs has long been investigated by CS educators, especially with the rise of "homework help" websites over the past decade, and recently with AI tools like ChatGPT. The main detection approach has long been flagging similar submission pairs. Modern cheating, like hiring contractors or using ChatGPT, may not yield such similarity. And, cases based on similarity alone may be weak. Thus, over the past several years, building on logs from an online program auto-grader (zyBooks), we developed additional "cheating concern metrics": points rate, style anomalies, style inconsistencies, IP address anomalies, code replacements, and initial copying. Most are defined not only for one programming assignment but also across a set of assignments. The metrics can help catch more kinds of cheating, provide more compelling evidence of cheating, reduce false cheating accusations based on similarity alone, and help instructors focus their limited cheat-detection time on the most egregious cases. We describe the techniques, and our experiences (via our own Python scripts and a commercial tool) for several terms, showing benefits of having more metrics than just similarity. Of 30 cheating cases over 3 terms and 300 students, most were based on metrics beyond similarity, all students admitted, none later contested, and time per student was only 1-2 hours (far less than previously). Our goal is to prevent cheating in the first place, by reducing opportunity via strong detection tools, as part of a multi-faceted approach to having students truly learn and stay out of trouble.

2 March 2023
Verified
Conference Paper
Open access
FVFrank Vahid
Frank Vahid
KDKelly Downey
Kelly Downey
APAshley Pang
Ashley Pang
CGChelsea Gordon
Chelsea Gordon
Cheating in introductory programming classes (CS1) is a well-known problem. Various methods have been suggested to reduce cheating, but many are time-consuming, resource intensive, or don't scale to large classes. We introduced a class intervention having 6 low-effort commonly-suggested methods to reduce cheating: (1) Discussing academic integrity for 20-30 minutes, several weeks into the term, (2) Requiring an integrity quiz with explicit do's and don'ts, (3) Allowing students to retract program submissions, (4) Reminding students mid-term about integrity and consequences of getting caught, (5) Showing instructor tools in class (including a similarity checker, statistics on time spent, and access to a student's full coding history), (6) Normalizing help and pointing students to help resources. Via manual evaluation of similarity checker results on 7 held-constant labs with one instructor teaching 100-student sections, for two pre-intervention and two intervention sections, suspected-cheating reduced 62% (30.5% down to 11.5%). Because manual evaluation could be biased and is time consuming, we developed two automated coding-behavior metrics per lab -- time spent programming, and % of students with highly-similar code -- that may suggest how much cheating is happening. Time spent increased by 56% (7 min to 10.9 min), and % of students with highly-similar code dropped 48% (38.5% to 20%). We later repeated the intervention with a second instructor and different labs and achieved similar (in fact, even stronger) results, with time rising 84% (13 min to 24 minutes) and % dropping 66% (55.5% to 19%). All findings were statistically significant with p < 0.0001.

3 July 2024
Verified
Conference Paper
Open access
BDBenjamin Denzler
Benjamin Denzler
FVFrank Vahid
Frank Vahid
APAshley Pang
Ashley Pang
MSMariam Salloum
Mariam Salloum
Student cheating on at-home programming assignments is a well- known problem. A key contributor is externally-obtained solutions from websites, contractors, and recently generative AI. In our experience, such externally-obtained solutions often use coding styles that depart from a class' style, which we call "style anomalies," such as using untaught or advanced constructs like pointers or ternary operators, or having different indenting or brace usage from the class style. We developed a tool to auto-count style anomalies. For six labs across four terms in 2021-2022, and 50 sampled students per lab, we found 18% of submissions on average had unusually-high style anomaly counts. Importantly, 8% of submissions on average had a high style anomaly count but were not flagged by a similarity checker, meaning 8% of submissions are suspicious but might have been missed if using similarity checking alone. We repeated a similar analysis for Spring 2023 when generative AI (ChatGPT) was gaining popularity, and the numbers rose to 26% and 18%, respectively. Detailed investigations by instructors led to a majority (but not all) high style anomaly submissions being deemed cheating. Even for high-similarity submissions, counting style anomalies can help instructors focus investigations on the most-likely cheating cases, and can strengthen cases sent to student conduct offices. With the rise of externally-obtained solutions from websites, contractors, and generative AI, counting style anomalies may become an increasingly important complement to similarity checking; in fact, it is now the primary cheat-detection tool in our CS1 at a large state university, with similarity secondary.

12 February 2025
Verified
Conference Paper
Open access
SSShirin Haji Amin Shirazi
Shirin Haji Amin Shirazi
APAshley Pang
Ashley Pang
AKAllan Knight
Allan Knight
MSMariam Salloum
Mariam Salloum
FVFrank Vahid
Frank Vahid
The ubiquitous use of online tools, contractors and homework sites, has made plagiarism a concerning topic in computer science education. With the introduction of ChatGPT, it poses a threat now more than ever. Many cheating detection tools, such as similarity checkers and style anomaly checkers, help instructors decide whether a student has plagiarized. However, these are not scalable to large classes. Similarity tools can produce high rates of suspected cheating and thus ineffectively use an instructor's time in weeding out the actual cheating cases, especially in the early weeks of CS courses where programs can be small and student solutions can be very similar. We developed a new approach using outlier detection to filter inconsistent performers based on their lab scores throughout the course and their midterm exam scores. Instructors can then manually analyze a manageable amount of students even with large class sizes. We performed our experiment on two large course offerings of CS1 (a total of 177 students) using our algorithm and compared it to a manual analysis performed by an experienced CS1 instructor. The detection approach identified 11 students in the first offering (Winter 2019) and 12 students in the second offering (Spring 2023). With an average precision of 83%, our tool produces a list of concerning students with high precision. This significantly helps teachers efficiently allocate their time and pursue cheating early in the term in order to address and prevent further issues.

29 June 2023
Verified
Conference Paper
Open access
APAshley Pang
Ashley Pang
FVFrank Vahid
Frank Vahid
Similarity checking is a common approach for detecting cheating in programming courses. A known limitation is high rates of similar pairs for programs lacking variability in possible solutions, especially for small programs. We experienced this issue in our CS1 course, where similarity checking in early weeks yielded many highly-similar pairs, many of which were not likely due to copying. Yet, we wish to catch copying students early, so that we can intervene and help those students avoid developing copying habits that may cause them trouble later. Our approach is to modify the program specifications to include variability-inducing requirements, namely places in the specifications where students make choices in their solutions, where different choices reduce the similarity scores. Those variability-inducing requirements are intentionally designed to avoid making the problem much harder for students. Examples of variability-inducing requirements include adding requirements to check for invalid input, or counting items. Such requirements have many different possible ways of implementing each. Essentially, variability-inducing requirements decrease the odds that two students would submit programs scored as highly-similar by a similarity checker, even for small programs. For 5 programs in our CS1 course, we added some variability-inducing requirements. Compared to an earlier term, the similarity checker's highly-similar-pairs rate dropped from 52% to 20% on average. Students' scores stayed the same from 98% to 96%, though time did increase from 18 min to 31 min on average. Adding such requirements helps instructors to do similarity detection and perform early interventions if desired.