In this paper, simulation experiments were conducted to study the response of phytoplankton biomass and community composition to the influence of polymetallic nodules and sediment at four stations in the western Pacific in 2021. Chlorophyll a, pico-phytoplankton cell abundance, and metal concentration were measured before and after 24 h of deck incubation. The results show that there were three different patterns of response, namely, restrained, stimulated, and unaffected patterns. The restrained pattern appeared in the filtered treatments at station Incub.01, and the stimulated pattern appeared in the unfiltered treatments at station Incub.02. The response of the phytoplankton was not detectable at stations Incub.03 and 04. Regardless, positive and negative responses were found in the dominant pico-phytoplankton group—Prochlorococcus—and with slight variation in Synechococcus. The concentration of manganese varied among the treatments compared to that of iron and other metals. The factors affecting the growth of the phytoplankton in this study were metal concentrations and turbidity. The phytoplankton biomass baseline may also have played an important role: the lower the biomass, the higher the growth rate. This study proved that deep-sea polymetallic nodule mining will have a specific impact on surface phytoplankton biomass, but turbidity and particle retention time could be important factors in mitigating the extent of the impact.