The open-circuit voltage (Voc) of polymer:fullerene bulk heterojunction solar cells is determined by the interfacial charge-transfer (CT) states between polymer and fullerene. Fourier-transform photocurrent spectroscopy and electroluminescence spectra of several polymer:fullerene blends are used to extract the relevant interfacial molecular parameters. An analytical expression linking these properties to Voc is deduced and shown to be valid for photovoltaic devices comprising three commonly used conjugated polymers blended with the fullerene derivative [6,6]-phenyl-C61-butyric acid methyl ester (PCBM). Voc is proportional to the energy of the CT states ECT. The energetic loss q$\Delta${}V between ECT and qVoc vanishes when approaching 0 K. It depends linearly on T and logarithmically on illumination intensity. Furthermore q$\Delta${}V can be reduced by decreasing the electronic coupling between polymer and fullerene or by reducing the nonradiative recombination rate. For the investigated devices we find a loss q$\Delta${}V of $\sim${}0.6$\hskip0.3em${}eV at room temperature and under solar illumination conditions, of which $\sim${}0.25$\hskip0.3em${}eV is due to radiative recombination via the CT state and $\sim${}0.35$\hskip0.3em${}eV is due to nonradiative recombination.