In the real pasture environment, the non-contact estimation of sheep live weight and carcass weight faces challenges such as hair interference, incomplete modal information and difficulty in edge deployment. In this paper, a top-view multi-modal perception and lightweight fusion modeling framework is proposed. The synchronously collected back RGB images, depth maps and point clouds are used as inputs. After unified calibration, time synchronization and standardization of the region of interest, they are sent to the lightweight multi-modal fusion network LMF-Net. The network uses modal-specific coding, multi-scale feature integration and low-rank multi-modal fusion, and introduces a wool thickness adaptive correction module to suppress the geometric deviation caused by thick wool quilts. A total of 3757 valid samples of 148 Hu sheep, including 46 real pasture data sets with carcass weight labels, LMF-Net achieved MAE = 1.03 kg, RMSE = 1.28 kg, R2 = 0.890 for LW task, and MAE = 1.03 kg, RMSE = 1.18 kg, R2 = 0.801 for CW task. WTAC reduces the prediction errors of LW and CW by about 20 % and 17 %, respectively, on high wool index and high surface roughness samples. The number of model parameters is only 5.4 M, and the single-frame inference delay is 28.3 ms, which is significantly better than a variety of single-mode, dual-mode and representative multi-mode methods. The system further integrates the front end of the small program to realize the result display and historical record management, forming a closed loop of ' acquisition-reasoning-output ' application. This framework achieves a good balance between accuracy, robustness and lightweight deployment, and provides a technical solution for non-contact measurement of sheep production performance in intelligent animal husbandry scenarios.