Machining industry has been evolving toward implementation of automation into the processes for higher productivity and efficiency. Although many studies have been conducted in the past to develop intelligent monitoring systems in various application scenarios of machining processes, most of them just focused on cutting tools without considering the influence of the nonuniform hardness of workpiece material. This study develops a compact, reliable, and cost-effective tool condition monitoring (TCM) system to detect the cutting tool wear in machining of the workpiece material with hardness variation. The generated audible sound signals during the machining process are analyzed by state-of-the-art artificial intelligent techniques, support vector machine (SVM) and convolutional neural network (CNN), to predict the tool wear and the hardness variation of the workpiece. A four-level classification model is developed for the system to detect the tool wear condition based on the width of the flank wear land and the hardness variation of the workpiece. This study also involves the comparative analysis between two employed artificial intelligent techniques to evaluate the performance of the model in prediction. The proposed TCM system has shown a high prediction accuracy in detecting the tool wear from the audible sound into the proposed multiclassification wear level in end milling of the nonuniform hardened workpiece.