The integration of photovoltaic (PV) and electric vehicle (EV) charging in residential buildings has increased in recent years. At high latitudes, both pose new challenges to the residential power systems due to the negative correlation between household load and PV power production and the increase in household peak load by EV charging. EV smart charging schemes can be an option to overcome these challenges. This paper presents a distributed and a centralized EV smart charging scheme for residential buildings based on installed photovoltaic (PV) power output and household electricity consumption. The proposed smart charging schemes are designed to determine the optimal EV charging schedules with the objective to minimize the net load variability or to flatten the net load profile. Minimizing the net load variability implies both increasing the PV self-consumption and reducing the peak loads. The charging scheduling problems are formulated and solved with quadratic programming approaches. The departure and arrival time and the distance covered by vehicles in each trip are specifically modeled based on available statistical data from the Swedish travel survey. The schemes are applied on simulated typical Swedish detached houses without electric heating. Results show that both improved PV self-consumption and peak load reduction are achieved. The aggregation of distributed smart charging in multiple households is conducted, and the results are compared to the smart charging for a single household. On the community level, both results from distributed and centralized charging approaches are compared.