The variations in the internal tide (IT) and near‐inertial waves (NIWs) that occurred in the summer of 2006 over the continental shelf of the northern South China Sea were observed using three bottom‐mounted moorings equipped with acoustic Doppler current profilers. Due to the strong nonlinear interaction, large portions (~40–60%) of the semidiurnal IT energy at the continental shelf were found to be incoherent; this proportion gradually increased from the shelf break to the offshore region. More importantly, our observations showed that parametric subharmonic instability (PSI) can enhance the f‐band energy in the shallow water (~100 m) on the continental shelf. Using plane‐wave fitting and the bicoherence spectrum, we confirmed that the intensified NIWs and the observed D2‐f waves are nonlinear coupled waves derived from the semidiurnal internal tidal PSI. These two subharmonic waves were observed in the near field of a strong semidiurnal IT generation site at the shelf break. By tracking the M2‐ray path, we verified that the PSI occurred in the pycnocline in the path of the reflected wave beam that was characterized by strong nonlinear instabilities and it could be modulated by mesoscale eddies in this shallow shelf water.