Recently, deep neural networks (DNNs) have been widely used in various fields such as autonomous vehicles and smart homes. Since these DNNs can be directly implemented on edge devices, they offer advantages such as real-time processing in low-power and low-bandwidth environments. However, deployment of DNNs in embedded systems, including edge devices, exposes them to threats such as fault injection attacks. In this paper, we introduce two methods to induce misclassifications by using clock and voltage glitch-based fault attacks in devices where DNN models are executed. As a result of experiments on a microcontroller implemented with a DNN for image classification, we show that glitch injection attacks can lead with high probability to serious misclassifications. Furthermore, we propose a countermeasure to glitch attacks on the softmax function, and confirm that this method is effective in preventing misclassifications.