Significant progress has been achieved by using deep learning technology for the semantic segmentation of airborne laser scanning (ALS) point clouds. However, there are still challenges in effectively capturing contextual information and fusing network multilevel features to improve ALS point cloud segmentation performance. In this letter, we propose a lightweight bidirectional feature aggregation and adaptive fusion network. First, we propose a novel adaptive bidirectional feature aggregation module to adaptively aggregate multi-scale local features, effectively expanding the receptive field. After that, we introduce a feature adaptive fusion module between up-sampling and down-sampling in the same layer to explore more comprehensive multilevel features and finer contextual information. Experiments on ISPRS 3D, GML, and LASDU datasets verify the superior segmentation performance of the proposed method with mF1 of 72.83% on ISPRS 3D, 76.12% on GML, and 78.95% on LASDU.