Okra (Abelmoschus esculentus (L.) Moench), a healthy vegetable, is widely spread in tropical and subtropical areas. Previous studies have proven that okra pods possess anti-fatigue activity, and the aim of this research is to clarify the anti-fatigue constituents. To achieve this, we divided okra pods (OPD) into seeds (OSD) and skins (OSK), and compared the contents of total polysaccharides, total polyphenols, total flavonoids, isoquercitrin, and quercetin-3-O-gentiobiose and the antioxidant activity in vitro and anti-fatigue activity in vivo between OSD and OSK. The contents of total polyphenols and total polysaccharides were 29.5% and 14.8% in OSD and 1.25% and 43.1% in OSK, respectively. Total flavonoids, isoquercitrin and quercetin-3-O-gentiobiose (5.35%, 2.067% and 2.741%, respectively) were only detected in OSD. Antioxidant assays, including 1-diphenyl-2-picrylhydrazyl (DPPH) scavenging, ferric reducing antioxidant power (FRAP) and reducing power test, and weight-loaded swimming test showed OSD possessed significant antioxidant and anti-fatigue effects. Moreover, biochemical determination revealed that that anti-fatigue activity of OSD is caused by reducing the levels of blood lactic acid (BLA) and urea nitrogen (BUN), enhancing hepatic glycogen storage and promoting antioxidant ability by lowering malondialdehyde (MDA) level and increasing superoxide dismutase (SOD) and glutathione peroxidase (GSH-PX) levels. These results proved okra seeds were the anti-fatigue part of okra pods and polyphenols and flavonoids were active constituents.