In order to clarify ultra-low-frequency (ULF) seismomagnetic phenomena, a sensitive geomagnetic network was installed in Kanto, Japan since 2000. In previous studies, we have verified the correlation between ULF magnetic anomalies and local sizeable earthquakes. In this study, we use Molchan’s error diagram to evaluate the potential earthquake precursory information in the magnetic data recorded in Kanto, Japan during 2000–2010. We introduce the probability gain (PG′) and the probability difference (D′) to quantify the forecasting performance and to explore the optimal prediction parameters for a given ULF magnetic station. The results show that the earthquake predictions based on magnetic anomalies are significantly better than random guesses, indicating the magnetic data contain potential useful precursory information. Further investigations suggest that the prediction performance depends on the choices of the distance (R) and size of the target earthquake events (Es). Optimal R and Es are about (100 km, 10^8.75) and (180 km, 10^8.75) for Seikoshi (SKS) station in Izu and Kiyosumi (KYS) station in Boso, respectively.