In this study, the extraction conditions for selenium-enriched rape protein (SEP) were optimized by applying a response surface methodology (RSM) and artificial neural network (ANN) model, and then, the optimal conditions were obtained using a genetic algorithm (GA). Then, the antioxidant power of the SEP was examined by using the DPPH, ABTS, and CCK-8 (Cell Counting Kit-8), and its anticancer activities were explored by conducting a cell migration test. The results showed that compared with the RSM model, the ANN model was more accurate with a higher determination coefficient and fewer errors when it was applied to optimize the extraction method. The data obtained for SEP using a GA were as follows: the extraction temperature was 59.4 °C, the extraction time was 3.0 h, the alkaline concentration was 0.24 mol/L, the liquid-to-material ratio was 65.2 mL/g, and the predicted content of protein was 58.04 mg/g. The protein was extracted under the conditions obtained by the GA; the real content of protein was 57.69 mg/g, and the protein yield was 61.71%. Finally, as the concentration of the selenium-containing protein increased, it showed increased ability in scavenging free radicals and was influential in inhibiting the proliferation and migration of HepG2 cells.