Light scattering at nanoparticles and molecules can be dramatically enhanced in the 'hot spots' of optical antennas, where the incident light is highly concentrated. Although this effect is widely applied in surface-enhanced optical sensing, spectroscopy and microscopy, the underlying electromagnetic mechanism of the signal enhancement is challenging to trace experimentally. Here we study elastically scattered light from an individual object located in the well-defined hot spot of single antennas, as a new approach to resolve the role of the antenna in the scattering process. We provide experimental evidence that the intensity elastically scattered off the object scales with the fourth power of the local field enhancement provided by the antenna, and that the underlying electromagnetic mechanism is identical to the one commonly accepted in surface-enhanced Raman scattering. We also measure the phase shift of the scattered light, which provides a novel and unambiguous fingerprint of surface-enhanced light scattering.