This article proposes a feature self-selection and sequence integration network, namely FASSI-Net, for medical image classification, which can extract representative deep features and contextual semantic information. In this network, FASSI-Net uses a new feature selection and integration module (FSIM) to compress the depth features, which uses a sequence model to replace the Flatten layer. This strategy introduces two sets of multi-scale convolutions, where a cross-attention mechanism assigns two sets of weights (i.e., vertical and horizontal weights) to each convolution. We then calculate the Euclidean distance between different scale feature points to measure the correlation between them. Specifically, the feature points are divided into useful features and redundant features. In addition, a feature dimension compression (CRI) module is constructed to reconstruct the redundant feature structure, and the residual structure is used to extract the representative features from the redundant features. Meantime, a sequence model is introduced to compress the deep features and obtain the context relationship between feature points. Experimental results on three datasets show that the proposed method significantly outperforms previous methods.