Behavior-based malware analysis is an important technique for automatically analyzing and detecting malware, and it has received considerable attention from both academic and industrial communities....

Hybrid fuzzing is a popular software testing technique that combines random fuzzing with concolic execution. It is widely used in the security domain known for its ability to find deeply hidden vulnerabilities and reach high code coverage. Hybrid fuzzing is based on negating branches in the execution path of a specific input to generate new test cases. However, due to numerous inputs and related branches, it does not show the best of its effectiveness without input and branch selection methods. In this paper, we systematically analyze the branch scheduling problem in the internal attributes of hybrid fuzzing, focusing on the synchronization mechanism. To solve the problems, we propose the Selective Hybrid Fuzzing (SHF) approach with branch scheduling based on binary instrumentation. There are two major parts to the SHF approach: (1) we propose a critical branch selection algorithm to select critical branches by three metrics: hit accuracy, solvability, and complexity; (2) we propose a priority score calculation algorithm to select inputs by the number of critical branches. With the SHF approach, we choose only the branches that can be negated to generate new coverage, instead of repeatedly executing the same branches and generating duplicates of inputs. We implement a hybrid fuzzer called SHFuzz with our SHF approach and compare it with the state-of-the-art hybrid fuzzer QSYM. In the evaluation, SHFuzz outperforms QSYM in 20 real-world applications from the Google Fuzzer Test Suite and other program suites in a 12 h test. On average, SHFuzz achieves 8.40% more code coverage and 100 more unique crashes in each application. Our work also finds existing vulnerabilities 7.85× faster than QSYM. We also find new bugs by SHFuzz, which QSYM fails to find. Our evaluation shows that the selective hybrid fuzzing approach can reduce the number of branches executed in concolic execution, enhancing hybrid fuzzing on code coverage and bug finding capabilities.

A large number of IoT devices access the Internet. While enriching our lives, IoT devices bring potential security risks. Device identification is one effective way to mitigate security risks and manage IoT assets. Typical identification algorithms generally separate data capture and target identification into two parts. As a result, it is inefficient and coarse-grained to evaluate the results only once the identification process is complete and then adjust the data capture strategy afterward. To solve this problem, we propose a fine-grained probe-scheduling approach based on information feedback. First, we model the probe surface as three layers for IoT devices and define their relationships. Then, we improve the policy gradient algorithm to optimize the probe policy and generate the optimal probe sequence for the target device. We implement a prototype system and evaluate it on 53,000 IoT devices across various categories to show its wide applicability. The results indicate that our approach can achieve success rates of 96.89%, 93.43%, and 83.71% for device brand, model, and firmware version, respectively, and reduce the identification time by 55.96%.

Network protocol software is usually characterized by complicated functions and a vast state space. In this type of program, a massive number of stateful v