Object detection in deep learning provides a viable solution for detecting crop-pest-infected regions. However, existing rectangle-based object detection methods are insufficient to accurately detect the shape of pest-infected regions. In addition, the method based on instance segmentation has a weak ability to detect the pest-infected regions at the edge of the leaves, resulting in unsatisfactory detection results. To solve these problems, we constructed a new polygon annotation dataset called PolyCorn, designed specifically for detecting corn leaf pest-infected regions. This was made to address the scarcity of polygon object detection datasets. Building upon this, we proposed a novel object detection model named Poly-YOLOv8, which can accurately and efficiently detect corn leaf pest-infected regions. Furthermore, we designed a loss calculation algorithm that is insensitive to ordering, thereby enhancing the robustness of the model. Simultaneously, we introduced a loss scaling factor based on the perimeter of the polygon, improving the detection ability for small objects. We constructed comparative experiments, and the results demonstrate that Poly-YOLOv8 outperformed other models in detecting irregularly shaped pest-infected regions, achieving 67.26% in mean average precision under $0.5$ threshold ($\mathrm{mA}{\mathrm{P}}_{50}$) and 128.5 in frames per second (FPS).