Weed management is becoming increasingly important for sustainable crop production. Weeds cause an average yield loss of 11.5% billion in Pakistan, which is more than PKR 65 billion per year. A real-time laser weeding robot can increase the crop’s yield by efficiently removing weeds. Therefore, it helps decrease the environmental risks associated with traditional weed management approaches. However, to work efficiently and accurately, the weeding robot must have a robust weed detection mechanism to avoid physical damage to the targeted crops. This work focuses on developing a lightweight weed detection mechanism to assist laser weeding robots. The weed images were collected from six different agriculture farms in Pakistan. The dataset consisted of 9000 images of three crops: okra, bitter gourd, sponge gourd, and four weed species (horseweed, herb paris, grasses, and small weeds). We chose a single-shot object detection model, YOLO5. The selected model achieved a mAP of 0.88@IOU 0.5, indicating that the model predicted a large number of true positive (TP) with much less prediction of false positive (FP) and false negative (FN). While SSD-ResNet50 achieved a mAP of 0.53@IOU 0.5, the model predicted fewer TP with significant outcomes as FP or FN. The superior performance of the YOLOv5 model made it suitable for detecting and classifying weeds and crops within fields. Furthermore, the model was ported to an Nvidia Xavier AGX standalone device to make it a high-performance and low-power computation detection system. The model achieved an FPS rate of 27. Therefore, it is highly compatible with the laser weeding robot, which takes approximately 22.04 h at a velocity of 0.25 feet per second to remove weeds from a one-acre plot.