Background/Objectives: Acute pancreatitis (AP) is a significant inflammatory pancreatic disease with high morbidity and mortality rates that requires early and accurate diagnosis. In this study, a deep learning-based classification system is developed and evaluated for the automatic classification of AP and normal pancreas from contrast-enhanced CT images, with a focus on patient-level assessment to enhance clinical applicability. Methods: A study-specific dataset is created for the study in CT images from 183 patients (103 normal and 80 with AP). To prevent data leakage and objectively evaluate model performance, the dataset is divided into training and test sets based on patient-level data. Convolutional neural network (CNN)-based architectures, such as ResNet50, EfficientNet, and ConvNeXtV2, are compared with Transformer-based architectures, such as Swin Transformer (Swin) and Vision Transformer (ViT). Results: In slice-level analysis, all models achieve high performance. Swin shows the highest accuracy (84.06%), and ViT revealed the most balanced performance with an F1-score of 82.90%. In the more clinically significant patient-level evaluation, the ViT model outperforms all others with an accuracy of 89.19%, an F1-score of 86.67%, and an area under the curve (AUC) of 0.946. The ViT model’s high AUC and recall values demonstrate its ability to reliably distinguish between AP and normal pancreas classes, even under different threshold values. Conclusions: These results suggest that transformer-based architectures can extract stronger and more reliable feature representations from pancreatic CT images due to their capacity to model global contextual features. Furthermore, the patient-level evaluation approach enables the model to generate results that are more compatible with clinical decision-making processes, thereby enhancing its usability in real clinical settings. In conclusion, the proposed ViT-based approach is a promising method for diagnosing acute pancreatitis.