Pingchuan Ma

Pingchuan Ma

Auto-generated profile, based on publicly available information

Pingchuan Ma

Zhejiang University of Technology, Hangzhou, China

Research Focus & Metrics

Academic Career Length

12Years (2015-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 July 2020
Verified
Conference Paper
Closed access
PMPingchuan Ma
Pingchuan Ma
SWShuai Wang
Shuai Wang
JLJin Liu
Jin Liu
Natural language processing (NLP) models have been increasingly used in sensitive application domains including credit scoring, insurance, and loan assessment. Hence, it is critical to know that the decisions made by NLP models are free of unfair bias toward certain subpopulation groups. In this paper, we propose a novel framework employing metamorphic testing, a well-established software testing scheme, to test NLP models and find discriminatory inputs that provoke fairness violations. Furthermore, inspired by recent breakthroughs in the certified robustness of machine learning, we formulate NLP model fairness in a practical setting as (ε, k)-fairness and accordingly smooth the model predictions to mitigate fairness violations. We demonstrate our technique using popular (commercial) NLP models, and successfully flag thousands of discriminatory inputs that can cause fairness violations. We further enhance the evaluated models by adding certified fairness guarantee at a modest cost.

16 January 2024 in IEEE Transactions on Dependable and Secure Computing
Verified
Journal Article
Closed access
ZLZongjie Li
Zongjie Li
ZLZhibo Liu
Zhibo Liu
WWWai Kin Wong
Wai Kin Wong
PMPingchuan Ma
Pingchuan Ma
SWShuai Wang
Shuai Wang
In recent years, query-based static application security testing (Q-SAST) tools such as CodeQL have gained popularity due to their ability to codify vulnerability knowledge into SQL-like queries and search for vulnerabilities in the database derived from the software. The industry has made considerable progress in building Q-SAST tools, facilitating their integration into the continuous integration (CI) pipeline, and sustaining an active community. However, we do not have a systematic understanding of their vulnerability detection capability in comparison to conventional SAST tools. We conduct the first in-depth study of Q-SAST to demystify their C/C++ vulnerability detectability. Our study is conducted from three complementary aspects. We first use a synthetic CWE test suite and a real-world CVE test suite, totaling almost 30K programs with known CWE/CVE, to assess popular (commercial) Q-SAST and industry-leading SAST (requiring no queries). Then, we gather defect-fixing pull requests (PRs) since the release dates of three popular Q-SAST tools, characterizing historically-fixed defects and comparing them to pitfalls exposed in our CWE/CVE study. To enhance vulnerability detection, we design SAST-MT, a metamorphic testing framework to detect false positives (FPs) and false negatives (FNs) of Q-SAST. Findings of SAST-MT can be used to easily expose the root causes of Q-SAST's FPs and FNs. We summarize lessons from our study that can benefit both users and developers of Q-SAST.

27 May 2023 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
HWHuaijin Wang
Huaijin Wang
PMPingchuan Ma
Pingchuan Ma
SWShuai Wang
Shuai Wang
QTQiyi Tang
Qiyi Tang
SNSen Nie
Sen Nie
SWShi Wu
Shi Wu
Binary code similarity is the foundation of many security and software engineering applications. Recent works leverage deep neural networks (DNN) to learn a numeric vector representation (namely, embeddings ) of assembly functions, enabling similarity analysis in the numeric space. However, existing DNN-based techniques capture syntactic-, control flow-, or data flow-level information of assembly code, which is too coarse-grained to represent program functionality. These methods can suffer from low robustness to challenging settings such as compiler optimizations and obfuscations. We present sem2vec , a binary code embedding framework that learns from semantics . Given the control-flow graph (CFG), 34 pages. of an assembly function, we divide it into tracelets , denoting continuous and short execution traces that are reachable from the function entry point. We use symbolic execution to extract symbolic constraints and other auxiliary information on each tracelet. We then train masked language models to compute embeddings of symbolic execution outputs. Last, we use graph neural networks, to aggregate tracelet embeddings into the CFG-level embedding for a function. Our evaluation shows that sem2vec extracts high-quality embedding and is robust against different compilers, optimizations, architectures, and popular obfuscation methods including virtualization obfuscation. We further augment a vulnerability search application with embeddings computed by sem2vec and demonstrate a significant improvement in vulnerability search accuracy.

13 June 2023 in Proceedings of the ACM on Management of Data
Verified
Journal Article
Open access
PMPingchuan Ma
Pingchuan Ma
RDRui Ding
Rui Ding
SWShuai Wang
Shuai Wang
SHShi Han
Shi Han
DZDongmei Zhang
Dongmei Zhang
In light of the growing popularity of Exploratory Data Analysis (EDA), understanding the underlying causes of the knowledge acquired by EDA is crucial. However, it remains under-researched. This study promotes a transparent and explicable perspective on data analysis, called eXplainable Data Analysis (XDA). For this reason, we present XInsight, a general framework for XDA. XInsight provides data analysis with qualitative and quantitative explanations of causal and non-causal semantics. This way, it will significantly improve human understanding and confidence in the outcomes of data analysis, facilitating accurate data interpretation and decision making in the real world. XInsight is a three-module, end-to-end pipeline designed to extract causal graphs, translate causal primitives into XDA semantics, and quantify the quantitative contribution of each explanation to a data fact. XInsight uses a set of design concepts and optimizations to address the inherent difficulties associated with integrating causality into XDA. Experiments on synthetic and real-world datasets as well as a user study demonstrate the highly promising capabilities of XInsight.

8 February 2022 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
HWHuaijin Wang
Huaijin Wang
PMPingchuan Ma
Pingchuan Ma
YYYuanyuan Yuan
Yuanyuan Yuan
ZLZhibo Liu
Zhibo Liu
SWShuai Wang
Shuai Wang
SWShi Wu
Shi Wu
Binary code function search has been used as the core basis of various security and software engineering applications, including malware clustering, code clone detection, and vulnerability auditing. However, recognizing logically similar assembly functions remains a challenge. De facto binary code search tools rely on program structure-level information, including control flow and data flow graphs, extracted via standard program analysis techniques or deep neural networks (DNNs). However, DNN-based approaches capture lexical-, control structure-, or data flow-level information of binary code for representation learning, which can be too coarse-grained and does not faithfully denote program functionality. It can also suffer from low robustness to various challenging settings like compiler optimizations and obfuscations. This paper proposes a general solution to enhance the top-k ranked candidates of DNN-based binary code function search. The key idea is to design a low-cost and comprehensive equivalence check, which quickly exposes functionality deviations between the target function and its top-k matched functions. Functions failed this equivalence check can be shaved from the top-k list, whereas functions passing the check can be reconsidered to move ahead on the top-k ranked candidates, in a deliberate way. We design a practical and efficient equivalence check, named BinUSE, using under-constrained symbolic execution (USE). USE, a variant of symbolic execution, improves scalability by launching symbolic execution directly from function entry points and relaxing constraints over function parameters. It alleviates overhead incurred by path explosion and costly constraints. BinUSE is particularly designed to deliver a soundy function-level equivalence check, enhancing DNN-based binary code search by reducing its false alarms with low cost. Our evaluation shows that BinUSE can enable a general and effective enhancement of three DNN-based binary code search tools against challenges introduced by different compilers, optimizations, obfuscations, and architectures.

12 April 2024
Verified
Conference Paper
Open access
ZLZongjie Li
Zongjie Li
CWChaozheng Wang
Chaozheng Wang
PMPingchuan Ma
Pingchuan Ma
CLChaowei Liu
Chaowei Liu
SWShuai Wang
Shuai Wang
YLYang Liu
Yang Liu
Recent advances in large language models (LLMs) significantly boost their usage in software engineering. However, training a well-performing LLM demands a substantial workforce for data collection and annotation. Moreover, training datasets may be proprietary or partially open, and the process often requires a costly GPU cluster. The intellectual property value of commercial LLMs makes them attractive targets for imitation attacks, but creating an imitation model with comparable parameters still incurs high costs. This motivates us to explore a practical and novel direction: slicing commercial black-box LLMs using medium-sized backbone models. In this paper, we explore the feasibility of launching imitation attacks on LLMs to extract their specialized code abilities, such as "code synthesis" and "code translation." We systematically investigate the effectiveness of launching code ability extraction attacks under different code-related tasks with multiple query schemes, including zero-shot, in-context, and Chain-of-Thought. We also design response checks to refine the outputs, leading to an effective imitation training process. Our results show promising outcomes, demonstrating that with a reasonable number of queries, attackers can train a medium-sized backbone model to replicate specialized code behaviors similar to the target LLMs. We summarize our findings and insights to help researchers better understand the threats posed by imitation attacks, including revealing a practical attack surface for generating adversarial code examples against LLMs.