Protected agriculture is a field in which the use of automatic systems is a key factor. In fact, the automatic harvesting of delicate fruit has not yet been perfected. This issue has received a great deal of attention over the last forty years, although no commercial harvesting robots are available at present, mainly due to the complexity and variability of the working environments. In this work we developed a computer vision system (CVS) to automate the detection and localization of fruit in a tomato crop in a typical Mediterranean greenhouse. The tasks to be performed by the system are: (1) the detection of the ripe tomatoes, (2) the location of the ripe tomatoes in the XY coordinates of the image, and (3) the location of the ripe tomatoes’ peduncles in the XY coordinates of the image. Tasks 1 and 2 were performed using a large set of digital image processing tools (enhancement, edge detection, segmentation, and the feature’s description of the tomatoes). Task 3 was carried out using basic trigonometry and numerical and geometrical descriptors. The results are very promising for beef and cluster tomatoes, with the system being able to classify 80.8% and 87.5%, respectively, of fruit with visible peduncles as “collectible”. The average processing time per image for visible ripe and harvested tomatoes was less than 30 ms.