Single-tree diameter at breast height (DBH) and tree height (H) are fundamental parameters for forest inventory, forest structure characterization, and forest carbon stock estimation. However, single-source LiDAR data cannot simultaneously capture complete trunk and canopy structural information, limiting the accuracy of single-tree structural parameter estimation. To address this issue, a Bi-Directional Cross-Attention Fusion Network (BCAF-Net) is proposed to estimate DBH and H separately by integrating ground-based Simultaneous Localization and Mapping LiDAR (SLAM LiDAR) and Unmanned Aerial Vehicle LiDAR (UAV LiDAR) data. The framework employs a dual-branch encoder and a bidirectional cross-attention mechanism to establish cross-view structural relationships between trunk and canopy observations, enabling effective multi-source feature fusion. Experiments conducted at two urban forest sites demonstrated that BCAF-Net achieved the highest estimation accuracy, with RMSE of 0.82 cm for DBH and 0.91 m for H and corresponding R² values of 0.97 and 0.96, respectively. Furthermore, the model maintained stable performance under varying forest structural complexities, cross-site conditions, and tree species. These results demonstrate that cross-view structural interaction effectively exploits complementary information from SLAM LiDAR and UAV LiDAR data, thereby improving single-tree structural parameter estimation in complex forest environments.