Accurate colon polyp detection and segmentation in colonoscopy images remain a challenging task due to the variability in colon polyp appearance, size, shape, and location. In this paper, we propose EvoGrayNet, an automated evolutionary framework for colon polyp segmentation that facilitates detailed measurement of polyp boundaries, shape, size, and structural contrast. EvoGrayNet integrates a custom-designed feature extraction module with an evolutionary neural architecture search method to enhance spatial segmentation accuracy and model efficiency. First, we introduce a gray module that combines standard and depthwise separable convolutions, batch normalization, and dropout to capture both local details and global context in colonoscopy images. Second, a lightweight attention gate (AG) module is proposed to refine feature maps by focusing on low-contrast regions and improving localization accuracy. Third, we integrate the gray module and the AG module into a supernet along with a dilated feature extractor (DFE) and a feature recalibration (FR) module. The DFE captures multiscale spatial information using dilated convolutions, while the FR recalibrates channel importance for better feature representation. Finally, EvoGrayNet is optimized through an evolutionary process involving crossover and mutation, systematically identifying configurations that enhance boundary delineation accuracy. Performance is quantitatively assessed using the dice similarity coefficient (DSC), intersection over union (IoU), and the 95th percentile hausdorff distance (HD95), which together reflect both the precision of spatial measurement and boundary adherence. Experimental results show that EvoGrayNet outperforms 12 existing methods across four public colon polyp datasets, offering superior detection and segmentation accuracy, lower complexity, reduced FLOPS, faster inference speed, and improved generalization ability.