Mangosteen pericarps (MP) often end up as agricultural waste despite being rich in powerful natural antioxidants such as anthocyanins and xanthones. This study compared the effect of different drying processes and times on phenolic compounds and antioxidant activities of MP. Fresh MP were subjected to 36 and 48 h of freeze-drying (−44 ± 1 °C) and oven-drying (45 ± 1 °C), and 30 and 40 h of sun-drying (31 ± 3 °C). The samples were analyzed for anthocyanins composition, total phenolic content (TPC), total flavonoid content (TFC), antioxidant activities, and color characteristics. Analysis of liquid chromatography-mass spectrometry (LC-MS) with electrospray ionization identified two anthocyanins in MP: cyanidin-3-O-sophoroside and cyanidin-3-O-glucoside. Overall, the drying process, time, and their interactions significantly (p < 0.05) influenced the phenolic compounds, antioxidant activities, and color in MP extracts. Both freeze-drying after 36 h (FD36) and 48 h (FD48) possessed significantly (p < 0.05) higher total anthocyanins (2.1–2.2 mg/g) than other samples. However, FD36 was associated with significantly (p < 0.05) higher TPC (~94.05 mg GAE/g), TFC (~621.00 mg CE/g), and reducing power (~1154.50 μmol TE/g) compared to FD48. Moreover, FD36 is more efficient for industrial applications due to less time and energy consumption. Subsequently, obtained dried MP extracts could be further utilized as an alternative to synthetic food colorants.