In diagnosing sleep disorders, sleep stage classification is a very essential yet time-consuming process. Most of the existing state-of-the-art approaches rely on hand-crafted features and multi-modality polysomnography (PSG) data, where prior knowledge is compulsory and high computation cost can be expected. Besides, few studies are able to obtain high accuracy sleep staging using raw single-channel electroencephalogram (EEG). To overcome these shortcomings, this paper proposes an end-to-end framework with a deep neural network, namely SingleChannelNet, for automatic sleep stage classification based on raw single-channel EEG. The proposed model utilizes a 90s epoch as the textual input and employs two multi-convolution blocks and several max-average pooling layers to learn different scales of feature representations. To demonstrate the efficiency of the proposed model, we evaluate our model using different raw single-channel EEGs (C4/A1 and Fpz-Cz) on two different datasets (CC-SHS and Sleep-EDF datasets). Experimental results show that the proposed architecture can achieve better over-all accuracy and Cohen’s kappa (CCSHS: 90.2%-86.5%, Sleep-EDF: 86.1%-80.5%) compared with state-of-the-art approaches. Additionally, the proposed model can learn features automatically for sleep stage classification using different single-channel EEGs with distinct sampling rates from different datasets without using any hand-engineered features.