To effectively treat lung and colon cancer and save lives, early and accurate identification is essential. Conventional diagnosis takes a long time and requires the manual expertise of radiologists. The rising number of new cancer cases makes it challenging to process massive volumes of data quickly. Different machine learning approaches to the classification and detection of lung and colon cancer have been proposed by multiple research studies. However, when it comes to self-learning classification and detection tasks, deep learning (DL) excels. This paper suggests a novel DL convolutional neural network (CNN) model for detecting lung and colon cancer. The proposed model is lightweight and multi-scale since it uses only 1.1 million parameters, making it appropriate for real-time applications as it provides an end-to-end solution. By incorporating features extracted at multiple scales, the model can effectively capture both local and global patterns within the input data. The explainability tools such as gradient-weighted class activation mapping and Shapley additive explanation can identify potential problems by highlighting the specific input data areas that have an impact on the model’s choice. The experimental findings demonstrate that for lung and colon cancer detection, the proposed model was outperformed by the competition and accuracy rates of 99.20% have been achieved for multi-class (containing five classes) predictions.