Sarah Shah

Sarah Shah

Auto-generated profile, based on publicly available information

Sarah Shah

Electrical and Software Engineering, University of Calgary, 2129 Calgary, Alberta, Canada, T2N 1N4

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 July 2019
Verified
Conference Paper
Closed access
Yasaman AmannejadYasaman Amannejad
Yasaman Amannejad
SSSarah Shah
Sarah Shah
DKDiwakar Krishnamurthy
Diwakar Krishnamurthy
MWMea Wang
Mea Wang
Users and operators of cloud-based Spark clusters often require quick insights on how the execution time of an application is likely to be impacted by the resources allocated to the application, e.g., the number of Spark executor cores assigned, and the size of the data to be processed. Existing techniques typically require extensive prior executions of the application under various resource allocation settings and data sizes to obtain an accurate model. In this paper, we explore the accuracy of a model with less prior executions of the application. Such a model can be useful for situations where quick predictions are required and little cluster resources are available for building a model. We use logs from two executions of an application with small sample data and different resource settings and explore the accuracy of the predictions for other resource allocation settings and input data sizes.

24 August 2021 in IEEE Open Journal of the Computer Society
Verified
Journal Article
Open access
SSSarah Shah
Sarah Shah
Yasaman AmannejadYasaman Amannejad
Yasaman Amannejad
DKDiwakar Krishnamurthy
Diwakar Krishnamurthy
MWMea Wang
Mea Wang
A data analytics application submitted to a Spark cluster often has to finish executing by a specified deadline. To use cluster resources effectively, the key challenge is having the ability to gain quick insights on how the execution time of any given application is likely to be impacted by the resources allocated to the application, e.g., the number of Spark executor cores and the size of the input data. Such insights can be used to quickly estimate the required resources needed for the desired execution time. Our paper proposes an automated execution time estimation approach called PERIDOT that involves executing a given application under a fixed resource setting with two small subsets of its input data to offer fast, lightweight execution time predictions. It analyzes these two executions to estimate the internal dependencies of the application and combines them with knowledge of Sparks data partitioning mechanisms to derive an analytic model that can estimate execution times for other resource settings and input data sizes. Our results from a wide range of applications and multiple Spark clusters show that PERIDOT can accurately estimate the execution time of an application from limited historical data, and suggest the minimum amount of resources required to meet an execution deadline.

19 July 2021 in IEEE Access
Verified
Journal Article
Open access
SSSarah Shah
Sarah Shah
YAYasaman Amannejad
Yasaman Amannejad
DKDiwakar Krishnamurthy
Diwakar Krishnamurthy
Apache Spark is being increasingly used to execute big data applications on cluster computing platforms. To increase system utilization, cluster operators often configure their clusters such that multiple co-located applications can simultaneously share the resources of a cluster node. With resource sharing, applications can compete with each other for shared node resources thereby interfering with each other’s performance. Many Spark applications take a long time to execute. Performance interference from other applications can thus cause a Spark application to fail or take even longer time to execute thereby wasting cluster resources and frustrating users. This motivates the need for an automated technique that can detect interference quickly and also diagnose the root cause of the interference to facilitate mitigation of the problem. Most existing approaches are not designed to offer quick interference detection and diagnosis. For example, they typically require extensive training data for every application of interest under various possible input data sizes and resource allocations. In this paper, we systematically investigate the design of a Machine Learning (ML) based technique that addresses this open problem. We implement a tool called Diaspore that integrates our findings. We evaluate the tool with a diverse set of 13 Spark applications executing on a real cluster. Experimental results show that Diaspore requires only small scale training data, i.e., executions under small input sizes and resource allocations. Furthermore, our results show that the tool can offer accurate predictions for applications not present in the training data. Consequently, Diaspore reduces the training time needed to offer predictions. Finally, the feature engineering underlying Diaspore ensures that the tool can detect and diagnose interference quickly in an online manner by sampling only a small fraction of a long running application’s execution. This can allow cluster operators to mitigate interference in an agile manner.