Novelty detection is the identification (normal or anomalous) of test data differing from the training data (all are normal). It has gained significant attention in many fields, due to the infrequency and unpredictability of anomalies in reality. Most novelty detection methods focus on improving performance under single modal data source conditions. However, in complex realities, relying on single modal data often fails to capture the true state of the object under observation. Thus, these methods cannot always offer accurate and reliable decisions in novelty detection. Moreover, research on novelty detection under multi-modal conditions is relatively limited. Therefore, we propose MSENet, an end-to-end multi-modal semantic-enhanced network based on autoencoder for novelty detection, including the feature-extraction and semantic-mining modules, which can effectively reconstruct the multi-modal data belonging to a target class by bridging the semantic gaps between different modalities. In addition, we develop an L₁^* regularization method based on L₁-norm to select more representative features by promoting sparser weights during the optimization process. The results of the novelty detection experiment on two multi-modal datasets demonstrate that our method outperforms state-of-the-art methods. Furthermore, an ablation study and a regularization experiment verified the effectiveness of the MSENet and L₁^* regularization methods.