Seismic interferometry using ambient noise provides an effective approach for subsurface imaging through reconstructing passive virtual source (PVS) responses. Traditional crosscorrelation (CC) seismic interferometry relies on a uniform dense distribution of passive sources in the subsurface, which is often challenging in practice. The multidimensional deconvolution method (MDD) alleviates reliance on passive-source distribution, but requires wavefield decomposition of the original data. This is difficult to accurately achieve for uncorrelated noise sources, leading to the existence of non-physical artifacts in the reconstructed PVS data. To address this issue, this study proposes a method to improve the accuracy of PVS data reconstruction using an enhanced U-Net. This data-driven approach circumvents the challenge of noise wavefield decomposition encountered in the traditional MDD. By integrating a feature fusion module into U-Net, multi-scale sampling information is leveraged to improve the network’s ability to capture detailed PVS data features. The combination of active-source data constraints and the modified MDD further optimizes PVS data retrieval during training. Numerical tests show that the proposed method effectively recovers waveform information in PVS retrieval records with non-ideally distributed sources, suppressing coherent noise and false events. The reconstructed recordings have a clear advantage in the reverse time migration (RTM) imaging results, with strong generalization performance across various velocity models.