High-throughput phenotyping is playing an increasingly important role in many areas of agriculture. Breeders will use it to obtain values for the traits of interest so that they can estimate genetic value and select promising varieties; growers may be interested in having predictions of yield well in advance of the actual harvest. In most phenotyping applications, image analysis plays an important role, drastically reducing the dependence on manual labor while being non-destructive. An automatic phenotyping system combines a reliable acquisition system, a high-performance segmentation algorithm for detecting fruits in individual images, and a registration algorithm that brings the images (and the corresponding detected plants or plant components) into a coherent spatial reference frame. Recently, significant advances have been made in the fields of robotics, image registration, and especially image segmentation, which each individually have improved the prospect of developing a fully integrated automatic phenotyping system. However, so far no complete phenotyping systems have been reported for routine use in a production environment. This work catalogs the outstanding issues that remain to be resolved by describing a prototype phenotyping system for a production tomato greenhouse, for many reasons a challenging environment.