An increasing number of embedded systems include dedicated neural hardware. To benefit from this specialized hardware, deep learning techniques to discover malware on embedded systems are needed. This effort evaluated candidate machine learning detection techniques for distinguishing exploited from non-exploited RISC-V program behavior using execution traces. We first developed a dataset of execution traces containing Return Oriented Programming (ROP) exploitation on the RISC-V Instruction Set Architecture (ISA) and then developed several deep learning bidirectional Long Short-Term Memory (LSTM) models capable of distinguishing exploited traces from non-exploited traces, each using subsets of features from the execution traces. An objective of this effort was to evaluate which features (instruction addresses and immediate values) from an execution trace are application-specific, which features (opcodes and operands) are application-agnostic, and how these subsets of features affect model performance. Application-agnostic features allow a model to generalize its detection capability to detecting ROP in previously unseen applications. The model using opcode and operand sequences obtained 98.21% cross validation accuracy and 97.94% test accuracy. In contrast, a model using address values obtained 92.79% cross validation accuracy with 99.59% test set accuracy. This research also analyzed whether ROP exploitation significantly affects branch prediction; experimental evidence suggests that it does. Thus, branch prediction behavior could be a valuable feature in detecting ROP exploits.

Malware is an ever-present problem in the modern era and while detecting malware with AI has grown as a new field of exploration, current methods are not yet mature enough for widespread adoption in terms of speed and performance. Current methods largely focus on viewing malicious assembly as an image for detection, requiring a large amount of preprocessing and making network architectures inflexible. Preprocessing malware images to one size introduces additional time to predict and makes the task of prediction more difficult. We explore a novel method for transforming executable bytecode into a video rather than an image for classification with deep, time-distributed neural networks, achieving up to 98.74% testing accuracy on 9 classes of malware, and up to 99.36% testing accuracy on a balanced set of malicious vs. benign files. The network could also classify all malware in our dataset for a false positive rate of 13%, and was also found successful in classifying only parts of an input, as well as initial success in a 0-day scenario. The network only uses the executable code and no additional information to make predictions. We then explore methods for pruning and quantizing the network so that it may be more feasible for widespread implementation, including a novel pruning method we call Node-Distance pruning. Our model is found to be competitive to current works while remaining fast, lean and flexible.

We present our work on evaluating the usefulness of deep, convolutional neural networks (DNN) for classifying assembly or machine code as malicious or benign. Our results show that a DNN trained on a small dataset showed 95.1% accuracy in program classification. We also show a modified network can achieve 88% accuracy in classifying nine types of malware on a larger dataset, leaving room for future work to address variable length files.

Security for Cyber physical systems includes not only guaranteeing operational security of data they process, but preventing malicious alteration of their execution due to knowledge of their underlying structure. With the advent of software in the form of reprogrammable hardware descriptions, protection of field programmable units from malicious reverse engineering and subversion becomes more critical. We compare four different white-box transformation algorithms aimed at hindering adversarial reverse engineering by changing component and signal configurations within combinational logic programs. We present security and efficiency analysis for these techniques and show positive results for achieving measurable hiding of signal and component information.