Maintenance and repair of expressways are becoming increasingly important due to the growing frequency of their use. Accurate pavement crack information extraction helps with routine maintenance and reduces the risk of traffic accidents. The traditional 2D crack image detection method has limitations and cannot effectively obtain depth information. Three-dimensional crack extraction from 3D point cloud has become a new solution that can capture pavement crack information more comprehensively and accurately. However, the existing algorithms are not effective in the feature extraction of cracks due to the different and irregular shapes and sizes of pavement cracks and interference from the external environment. To solve this, a new method for detecting pavement cracks in point clouds, namely point attention net (PAN), is herein proposed. It uses a two-branch attention fusion module to focus on space and feature information in the cloud and capture features of crack points at different scales. It also uses the Poly Loss function to solve the imbalance of foreground and background points in pavement point cloud data. Experiments on the LNTU-RDD-LiDAR dataset were carried out to verify the effectiveness of the proposed method. Compared with the traditional method and the latest point cloud segmentation technology, the performance indexes of mIoU, Acc, F1, and Rec achieved significant improvement, reaching 75.4%, 91.5%, 75.4%, and 67.1%, respectively.