(1) Background: Accurate evaluation of colon capsule endoscopy videos plays a pivotal role in diagnosing gastrointestinal disorders. A primary step in this process is assessing the cleanliness of the area of interest to determine its admissibility. This study introduces a system designed to assist physicians in evaluating the colon cleanliness score of capsule endoscopy videos. (2) Methods: The system uses a TransUNet architecture, a customized loss function, and a low-effort labeling method to propose cleanliness scores for previously unseen videos. The proposed model was evaluated on a dataset of 52 capsule endoscopy videos. Agreement with physicians was measured using Cohen’s kappa statistic. (3) Results: The system achieved a Cohen’s kappa agreement of 0.586 with physicians, which is notably higher than the intra-observer variability observed, measured at 0.546. Additionally, the system can show the cleanliness evolution throughout the entire video, helping justify the proposed score. (4) Conclusions: The proposed system demonstrates improved agreement with physicians compared to inter-physician agreement, showing its potential to support the cleanliness evaluation process in colon capsule endoscopy. The ability to visualize the cleanliness evolution across the video enhances the transparency and interpretability of the suggested score.