Glaucoma is a common eye disease that causes optic nerve damage due to high intraocular pressure and eventually results in partial or permanent blindness if not detected timely. Hence, it is of utmost importance to detect glaucoma at an early stage for a better treatment plan. Recent years have witnessed significant efforts toward developing automated glaucoma classification methods using retinal fundus images. However, limited approaches have yet been explored for the detection of multiple stages of glaucoma. This is mainly due to the unavailability of large annotated datasets. Further, the presence of high inter-class similarities, subtle lesion size variations, and redundant features in the fundus images make the task more challenging. To address these issues, in this paper, we propose a novel cascaded attention-based network called CA-Net for efficient multi-stage glaucoma classification. A cascaded attention module (CAM) consisting of a triplet channel attention block and a spatial attention block is introduced on the top of a backbone network to learn feature dependencies along the channel, cross-channel, and spatial dimensions. The CAM helps in learning rich discriminative features from the key regions of the fundus image, thereby improving performance. Also, we establish a large multi-stage glaucoma (LMG) dataset and a binary glaucoma dataset, which contain 1582 and 623 fundus images, respectively. The experimental results on these datasets along with a publicly available dataset, show the superiority of our CA-Net over state-of-the-art methods. The Grad-CAM and Grad-CAM++ visualization results provide more insight into the performance of our proposed attention. Further, ablation studies are conducted to verify the effectiveness of each component of CA-Net.