Shane McIntosh

Shane McIntosh

Auto-generated profile, based on publicly available information

Shane McIntosh

University of Waterloo, Waterloo, Canada

Research Focus & Metrics

Academic Career Length

12Years (2015-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
18 January 2018 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
CTChakkrit Tantithamthavorn
Chakkrit Tantithamthavorn
SMShane McIntosh
Shane McIntosh
AHAhmed E. Hassan
Ahmed E. Hassan
KMKenichi Matsumoto
Kenichi Matsumoto
Defect prediction models---classifiers that identify defect-prone software modules---have configurable parameters that control their characteristics (e.g., the number of trees in a random forest). Recent studies show that these classifiers underperform when default settings are used. In this paper, we study the impact of automated parameter optimization on defect prediction models. Through a case study of 18 datasets, we find that automated parameter optimization: (1) improves AUC performance by up to 40 percentage points; (2) yields classifiers that are at least as stable as those trained using default settings; (3) substantially shifts the importance ranking of variables, with as few as 28% of the top-ranked variables in optimized classifiers also being top-ranked in non-optimized classifiers; (4) yields optimized settings for 17 of the 20 most sensitive parameters that transfer among datasets without a statistically significant drop in performance; and (5) adds less than 30 minutes of additional computation to 12 of the 26 studied classification techniques. While widely-used classification techniques like random forest and support vector machines are not optimization-sensitive, traditionally overlooked techniques like C5.0 and neural networks can actually outperform widely-used techniques after optimization is applied. This highlights the importance of exploring the parameter space when using parameter-sensitive classification techniques.

14 September 2015 in Empirical Software Engineering
Verified
Journal Article
Closed access
PDF file
YKYasutaka Kamei
Yasutaka Kamei
TFTakafumi Fukushima
Takafumi Fukushima
SMShane McIntosh
Shane McIntosh
KYKazuhiro Yamashita
Kazuhiro Yamashita
NUNaoyasu Ubayashi
Naoyasu Ubayashi
AHAhmed E. Hassan
Ahmed E. Hassan

12 April 2017 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
SMShane McIntosh
Shane McIntosh
YKYasutaka Kamei
Yasutaka Kamei
Just-In-Time (JIT) models identify fix-inducing code changes. JIT models are trained using techniques that assume that past fix-inducing changes are similar to future ones. However, this assumption may not hold, e.g., as system complexity tends to accrue, expertise may become more important as systems age. In this paper, we study JIT models as systems evolve. Through a longitudinal case study of 37,524 changes from the rapidly evolving Qt and OpenStack systems, we find that fluctuations in the properties of fix-inducing changes can impact the performance and interpretation of JIT models. More specifically: (a) the discriminatory power (AUC) and calibration (Brier) scores of JIT models drop considerably one year after being trained; (b) the role that code change properties (e.g., Size, Experience) play within JIT models fluctuates over time; and (c) those fluctuations yield over- and underestimates of the future impact of code change properties on the likelihood of inducing fixes. To avoid erroneous or misleading predictions, JIT models should be retrained using recently recorded data (within three months). Moreover, quality improvement plans should be informed by JIT models that are trained using six months (or more) of historical data, since they are more resilient to period-specific fluctuations in the importance of code change properties.

30 March 2021 in Empirical Software Engineering
Verified
Journal Article
Open access
PDF file
BCBodin Chinthanet
Bodin Chinthanet
RKRaula Gaikovina Kula
Raula Gaikovina Kula
SMShane McIntosh
Shane McIntosh
TITakashi Ishio
Takashi Ishio
AIAkinori Ihara
Akinori Ihara
KMKenichi Matsumoto
Kenichi Matsumoto
Security vulnerability in third-party dependencies is a growing concern not only for developers of the affected software, but for the risks it poses to an entire software ecosystem, e.g., Heartbleed vulnerability. Recent studies show that developers are slow to respond to the threat of vulnerability, sometimes taking four to eleven months to act. To ensure quick adoption and propagation of a release that contains the fix (fixing release), we conduct an empirical investigation to identify lags that may occur between the vulnerable release and its fixing release (package-side fixing release). Through a preliminary study of 231 package-side fixing release of npm projects on GitHub, we observe that a fixing release is rarely released on its own, with up to 85.72% of the bundled commits being unrelated to a fix. We then compare the package-side fixing release with changes on a client-side (client-side fixing release). Through an empirical study of the adoption and propagation tendencies of 1,290 package-side fixing releases that impact throughout a network of 1,553,325 releases of npm packages, we find that stale clients require additional migration effort, even if the package-side fixing release was quick (i.e., package-side fixing releasetypeSpatch). Furthermore, we show the influence of factors such as the branch that the package-side fixing release lands on and the severity of vulnerability on its propagation. In addition to these lags we identify and characterize, this paper lays the groundwork for future research on how to mitigate propagation lags in an ecosystem.

18 May 2018 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
KGKeheliya Gallaba
Keheliya Gallaba
SMShane McIntosh
Shane McIntosh
Continuous Integration (CI) is a popular practice where software systems are automatically compiled and tested as changes appear in the version control system of a project. Like other software artifacts, CI specifications require maintenance effort. Although there are several service providers like Travis CI offering various CI features, it is unclear which features are being (mis)used. In this paper, we present a study of feature use and misuse in 9,312 open source systems that use Travis CI. Analysis of the features that are adopted by projects reveals that explicit deployment code is rare—48.16% of the studied Travis CI specification code is instead associated with configuring job processing nodes. To analyze feature misuse, we propose Hansel—an anti-pattern detection tool for Travis CI specifications. We define four anti-patterns and Hansel detects anti-patterns in the Travis CI specifications of 894 projects in the corpus (9.60%), and achieves a recall of 82.76% in a sample of 100 projects. Furthermore, we propose Gretel—an anti-pattern removal tool for Travis CI specifications, which can remove 69.60% of the most frequently occurring anti-pattern automatically. Using Gretel, we have produced 36 accepted pull requests that remove Travis CI anti-patterns automatically.

5 July 2024 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
PCPartha Chakraborty
Partha Chakraborty
KAKrishna Kanth Arumugam
Krishna Kanth Arumugam
MAMahmoud Alfadel
Mahmoud Alfadel
MNMeiyappan Nagappan
Meiyappan Nagappan
SMShane McIntosh
Shane McIntosh
The impact of software vulnerabilities on everyday software systems is concerning. Although deep learning-based models have been proposed for vulnerability detection, their reliability remains a significant concern. While prior evaluation of such models reports impressive recall/F1 scores of up to 99%, we find that these models underperform in practical scenarios, particularly when evaluated on the entire codebases rather than only the fixing commit. In this paper, we introduce a comprehensive dataset ( Real-Vul ) designed to accurately represent real-world scenarios for evaluating vulnerability detection models. We evaluate DeepWukong, LineVul, ReVeal, and IVDetect vulnerability detection approaches and observe a surprisingly significant drop in performance, with precision declining by up to 95 percentage points and F1 scores dropping by up to 91 percentage points. A closer inspection reveals a substantial overlap in the embeddings generated by the models for vulnerable and uncertain samples (non-vulnerable or vulnerability not reported yet), which likely explains why we observe such a large increase in the quantity and rate of false positives. Additionally, we observe fluctuations in model performance based on vulnerability characteristics (e.g., vulnerability types and severity). For example, the studied models achieve 26 percentage points better F1 scores when vulnerabilities are related to information leaks or code injection rather than when vulnerabilities are related to path resolution or predictable return values. Our results highlight the substantial performance gap that still needs to be bridged before deep learning-based vulnerability detection is ready for deployment in practical settings. We dive deeper into why models underperform in realistic settings and our investigation revealed overfitting as a key issue. We address this by introducing an augmentation technique, potentially improving performance by up to 30%. We contribute (a) an approach to creating a dataset that future research can use to improve the practicality of model evaluation; (b) Real-Vul – a comprehensive dataset that adheres to this approach; and (c) empirical evidence that the deep learning-based models struggle to perform in a real-world setting.