With the continuous proliferation of renewable energy generation, distributed photovoltaic inverters operating at a maximum power point reduce the inertia of power systems, degrading system frequency stability and potentially causing severe oscillations in systems after being disturbed. The virtual synchronous generator (VSG) control method, which causes photovoltaic inverters to possess inertia and damping, now plays an important role in the field of distributed generation. However, while introducing the advantages of synchronous machines, problems with oscillations are also introduced and the stochastic fluctuation characteristic of photovoltaics results in the stochastic drifting of the operating point. This paper presents an adaptive controller parameter design method for a photovoltaic-VSG (PV-VSG) integrated power system. Firstly, a small-signal model of the PV-VSG is built and a state space model is deduced. Then, the small-signal stability and low frequency oscillation characteristics of the photovoltaic power generation system are analyzed. Finally, considering the limitations of system oscillations and the stochastic drifting of the operating point, a global optimization design method for controller parameters used to improve system stability is proposed. The time domain simulation shows that an optimized PV-VSG could provide sufficient damping in the case of photovoltaic power output changes across a wider range.