Advancements in AI have elevated speech recognition, with convolutional neural networks (CNNs) proving effective in processing spectrogram-transformed speech signals. CNNs, with lower parameters and higher accuracy compared to traditional models, are particularly efficient for deployment on storage-limited embedded devices. Artificial neural networks excel in predicting inputs within their expected output range but struggle with anomalies. This is usually harmful to a speech recognition system. In this paper, the neural network classifier for speech recognition is trained with a “negative branch” method, incorporating directional regularization with out-of-distribution training data, allowing it to maintain a high confidence score to the input within distribution while expressing a low confidence score to the anomaly input. It can enhance the performance of anomaly detection of the classifier, addressing issues like misclassifying the speech command that is out of the distribution. The result of the experiment suggests that the accuracy of the CNN model will not be affected by the regularization of the “negative branch”, and the performance of abnormal detection will be improved as the number of kernels of the convolutional layer increases.