Diabetes mellitus is a major health issue that has posed a significant challenge over the years. Gymnanthemum amygdalinum is a well-known plant that can be potentially used to treat this disease. Therefore, this study aimed to evaluate the inhibitory effect of its root, stem bark, leaves, and flower extracts on alpha-glucosidase using an in vitro inhibition assay to isolate the bioactive compounds and determine their levels in the samples. The air-dried plant parts were extracted by maceration using methanol. The results showed that the flower extract had the greatest inhibitory effect (IC50 47.29 ± 1.12 µg/mL), followed by the leaves, roots, and stem bark. The methanolic flower extract was further fractionated with different solvents, and the ethyl acetate fraction showed the strongest activity (IC50 19.24 ± 0.12 µg/mL). Meanwhile, acarbose was used as a positive control (IC50 73.36 ± 3.05 µg/mL). Characterization based on UV, 1H-, and 13C-NMR established that the ethyl acetate fraction yielded two flavonoid compounds, namely, luteolin and 2-(3,4-dihydroxyphenyl)-5,7-dihydroxy-3-methoxy-4H-chromen-4-on, which had IC50 values of 6.53 ± 0.16 µg/mL and 39.95 ± 1.59 µg/mL, respectively. The luteolin levels in the crude drug, methanolic extract, and ethyl acetate fraction were 3.4 ± 0.2 mg (0.3%), 32.4 ± 0.8 mg (3.2%), and 68.9 ± 3.4 mg (6.9%) per 1 g samples, respectively. These results indicated that the G. amygdalinum flower extract exerted potent inhibitory alpha-glucosidase activity.