Considering the high metastatic potential of colorectal cancer (CRC), the inhibition of metastasis is important for anti-CRC therapy. Agrimonia pilosa Ledeb (A. pilosa) is a perennial herbaceous plant that is widely distributed in Asia. The extracts of A. pilosa have shown diverse pharmacological properties, such as antimicrobial, anti-inflammatory, and antitumor activities. In the present study, the antimetastatic activity of A. pilosa was evaluated. Methanol extraction from the roots of A. pilosa was performed by high-performance liquid chromatography (HPLC) and 12 fractions were obtained. Among these, fraction 4 showed the most potent inhibitory effect on the migration of colon cancer cells. Using LC-HR MS analysis, quercetin and quercitrin were identified as flavonoids contained in fraction 4. Like fraction 4, quercetin and quercitrin effectively inhibited the migration and invasion of RKO cells. While the level of E-cadherin was increased, the levels of N-cadherin and vimentin were decreased by the same agents. Although they all activate the p38, JNK, and ERK signaling pathways, only SP600125, an inhibitor of the JNK pathway, specifically inhibited the effect of fraction 4, quercetin, and quercitrin on cell migration. An in vivo experiment also confirmed the antitumor activity of quercetin and quercitrin. Collectively, these results suggest that A. pilosa and its two flavonoids, quercetin and quercitrin, are candidates for the antimetastatic treatment of CRC.