Crack is one of the important pavement damages. The number and types of cracks are usually collected and inspected by human, but lately computer vision has shown great potential for automatically identifying pavement cracks. However, training a convolutional neural network (CNN) requires large amount of data. The limited number of crack images and the variety of crack image background put challenges for developing CNN model with high accuracy. This study proposes an intelligent method for crack pattern classification based on limited field images. The proposed method has a two-step data preprocessing. It first applied data augmentation method to enlarge the dataset, solving the data imbalance problem and providing more training data. Then a crack extraction method was applied to convert the original image into a binary back-and-white image. This step significantly reduced the input feature and simplified the model. Both of data preprocessing steps were designed to decrease the bias of the model. Then we performed a model selection and hyper-parameter tunning for CNN. We explored the application of AlexNet, SE-Net, and ResNet with a variety of configurations. The results show that the proposed data augmentation can enlarge the dataset significantly. The crack extraction also significantly increased the test accuracy. The ResNet with 50 layers has the highest test accuracy. With data augmentation, crack extraction and model selection combining together, the final approach can improve the test accuracy from 52.68% to 87.50% compared to the original limited images. The result suggests our proposed method works well to classify crack images of asphalt pavement with limited sample size.