Restoring vegetation in riparian zones is regarded as a best management practice in river restoration. Vegetation characteristics and diversity play a decisive role in maintaining ecological function in riparian zones. This study focuses on multi-scale distribution of herbaceous vegetation composition and diversity in riparian zones of three small–medium rivers in northern Zhejiang Province, China, through two years’ field investigations. Meanwhile, the main environmental and hydrological factors were analyzed by principal component analysis (PCA). The results indicated the following: (1) A total of 220 herbaceous plant species belonging to 55 families and 162 genera were recorded, with Poaceae (39 species, 17.73%) and Asteraceae (32 species, 14.55%) being the most abundant families. (2) Significant differences in riparian vegetation biomass and diversity were observed at both the river basin and river segment scales, in which upstream areas always showed higher richness and greater diversity of riparian vegetation. (3) The primary environmental factors influencing vegetation distribution varied with spatial scale: At the river scale, soil organic matter and water nitrogen were key factors affecting vegetation diversity, while riparian vegetation diversity was also influenced by water area. At the river segment scale, aquatic environmental factors exerted a more pronounced effect on vegetation diversity, with total phosphorus and nitrate nitrogen in water being the primary influencing factors. This research provides a theoretical basis for the restoration and sustainable management of riparian ecosystems in the study area and other similar regions.