Self-supervised learning representations of speech are successfully employed to language or dialect identification task. It usually adopts a self-supervised pre-trained model as the feature extractor, and adds a classifier to fine-tune on the target datasets. However, training a fine-tuned model from a pre-trained model leads to heavy time and storage cost. In this work, we attempt to build a competent dialect identification system using pre-trained self-supervised speech representations instead of fine-tuned input features. First, we simply fine-tune and compare several pre-trained self-supervised models to select an optimal framework as the speech representations extractor. Then, we introduce Transformers with sparse self-attention mechanism to model both global and local information of those representations. Finally, to further boost the performance, we explore some settings of our proposed method. We conduct experiments on the KeSpeech dataset. Experimental results show that, our method significantly outperforms the best known results, with cost average performance (Cavg) of 0.0911, equal error rate (EER) of 8.53%, and accuracy rate (Acc) of 80.38%. In addition, our proposed method obtains overall relative improvements compared to the fine-tuned one requiring less training time and computation.