Matthew C. Davis

Matthew C. Davis

Auto-generated profile, based on publicly available information

Matthew C. Davis

Carnegie Mellon University, USA

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
24 July 2023 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
MDMatthew C. Davis
Matthew C. Davis
EAEmad Aghayi
Emad Aghayi
TLThomas D. Latoza
Thomas D. Latoza
XWXiaoyin Wang
Xiaoyin Wang
BMBrad A. Myers
Brad A. Myers
JSJoshua Sunshine
Joshua Sunshine
A key goal of software engineering research is to improve the environments, tools, languages, and techniques programmers use to efficiently create quality software. Successfully designing these tools and demonstrating their effectiveness involves engaging with tool users—software engineers. Researchers often want to conduct user studies of software engineers to collect direct evidence. However, running user studies can be difficult, and researchers may lack solution strategies to overcome the barriers, so they may avoid user studies. To understand the challenges researchers face when conducting programmer user studies, we interviewed 26 researchers. Based on the analysis of interview data, we contribute (i) a taxonomy of 18 barriers researchers encounter; (ii) 23 solution strategies some researchers use to address 8 of the 18 barriers in their own studies; and (iii) 4 design ideas, which we adapted from the behavioral science community, that may lower 8 additional barriers. To validate the design ideas, we held an in-person all-day focus group with 16 researchers.

30 November 2023
Verified
Conference Paper
Open access
MDMatthew C. Davis
Matthew C. Davis
SCSangheon Choi
Sangheon Choi
SESam Estep
Sam Estep
BMBrad A. Myers
Brad A. Myers
JSJoshua Sunshine
Joshua Sunshine
In the United States alone, software testing labor is estimated to cost $48 billion USD per year. Despite widespread test execution automation and automation in other areas of software engineering, test suites continue to be created manually by software engineers. We have built a test generation tool, called NaNofuzz, that helps users find bugs in their code by suggesting tests where the output is likely indicative of a bug, e.g., that return NaN (not-a-number) values. NaNofuzz is an interactive tool embedded in a development environment to fit into the programmer's workflow. NaNofuzz tests a function with as little as one button press, analyses the program to determine inputs it should evaluate, executes the program on those inputs, and categorizes outputs to prioritize likely bugs. We conducted a randomized controlled trial with 28 professional software engineers using NaNofuzz as the intervention treatment and the popular manual testing tool, Jest, as the control treatment. Participants using NaNofuzz on average identified bugs more accurately (p < .05, by 30%), were more confident in their tests (p < .03, by 20%), and finished their tasks more quickly (p < .007, by 30%).

19 June 2025 in Proceedings of the ACM on Software Engineering
Verified
Journal Article
Open access
MDMatthew C. Davis
Matthew C. Davis
AWAmy Wei
Amy Wei
BMBrad A. Myers
Brad A. Myers
JSJoshua Sunshine
Joshua Sunshine
Software testing is difficult, tedious, and may consume 28%–50% of software engineering labor. Automatic test generators aim to ease this burden but have important trade-offs. Fuzzers use an implicit oracle that can detect obviously invalid results, but the oracle problem has no general solution, and an implicit oracle cannot automatically evaluate correctness. Test suite generators like EvoSuite use the program under test as the oracle and therefore cannot evaluate correctness. Property-based testing tools evaluate correctness, but users have difficulty coming up with properties to test and understanding whether their properties are correct. Consequently, practitioners create many test suites manually and often use an example-based oracle to tediously specify correct input and output examples. To help bridge the gaps among various oracle and tool types, we present the Composite Oracle, which organizes various oracle types into a hierarchy and renders a single test result per example execution. To understand the Composite Oracle’s practical properties, we built TerzoN, a test suite generator that includes a particular instantiation of the Composite Oracle. TerzoN displays all the test results in an integrated view composed from the results of three types of oracles and finds some types of test assertion inconsistencies that might otherwise lead to misleading test results. We evaluated TerzoN in a randomized controlled trial with 14 professional software engineers with a popular industry tool, fast-check, as the control. Participants using TerzoN elicited 72% more bugs (p < 0.01), accurately described more than twice the number of bugs (p < 0.01) and tested 16% more quickly (p < 0.05) relative to fast-check.

4 September 2025 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
MDMatthew C. Davis
Matthew C. Davis
SCSangheon Choi
Sangheon Choi
AWAmy Wei
Amy Wei
SESam Estep
Sam Estep
BMBrad A. Myers
Brad A. Myers
JSJoshua Sunshine
Joshua Sunshine
There is substantial diversity among testing tools used by software engineers. For example, fuzzers may target crashes and security vulnerabilities while Test sUite Generators (TUGs) may create high-coverage test suites. In the research community, test generation tools are primarily evaluated using metrics like bugs identified or code coverage. However, achieving good values for these metrics does not necessarily imply that these tools help software engineers efficiently develop effective test suites. To understand the test suite generation process, we performed a secondary analysis of recordings from a previously-published user study in which 28 professional software engineers used two tools to generate test suites for three programs with each tool. From these 168 recordings ( \(28\ users\times 2\ tools\times 3\ programs/tool\) ), we extracted a process model of test suite generation called TestLoop that builds upon prior work and systematizes a user’s test suite generation process for a single function into 7 steps. We then used TestLoop’s steps to describe 8 prior and 10 new recordings of users generating test suites using the Jest, Hypothesis, and NaNofuzz test generation tools. Our results showed that TestLoop can be used to help answer previously hard-to-answer questions about how users interact with test suite generation tools and to identify ways that tools might be improved.