Erosion is a relevant soil degradation factor in mountain agrosilvopastoral ecosystems that can be enhanced by the abandonment of agricultural land and pastures left to natural evolution. The on-site and off-site consequences of soil erosion at the catchment and landscape scale are particularly relevant and may affect settlements at the interface with mountain ecosystems. RUSLE (Revised Universal Soil Loss Equation) estimates of soil erosion consider, among others, the soil erodibility factor (K), which depends on properties involved in structure and aggregation. A relationship between soil erodibility and aggregation should therefore be expected. However, erosion may limit the development of soil structure; hence aggregates should not only be related to erodibility but also partially mirror soil erosion rates. The aim of the research was to evaluate the agreement between aggregate stability and erosion-related variables and to discuss the possible reasons for discrepancies in the two kinds of land use considered (forest and pasture). Topsoil horizons were sampled in a mountain catchment under two vegetation covers (pasture vs. forest) and analyzed for total organic carbon, total extractable carbon, pH, and texture. Soil erodibility was computed, RUSLE erosion rate was estimated, and aggregate stability was determined by wet sieving. Aggregation and RUSLE-related parameters for the two vegetation covers were investigated through statistical tests such as ANOVA, correlation, and regression. Soil erodibility was in agreement with the aggregate stability parameters; i.e., the most erodible soils in terms of K values also displayed weaker aggregation. Despite this general observation, when estimating K from aggregate losses the ANOVA conducted on the regression residuals showed land-use-dependent trends (negative average residuals for forest soils, positive for pastures). Therefore, soil aggregation seemed to mirror the actual topsoil conditions better than soil erodibility. Several hypotheses for this behavior were discussed. A relevant effect of the physical protection of the organic matter by the aggregates that cannot be considered in $K$ computation was finally hypothesized in the case of pastures, while in forests soil erodibility seemed to keep trace of past erosion and depletion of finer particles. A good relationship between RUSLE soil erosion rates and aggregate stability occurred in pastures, while no relationship was visible in forests. Therefore, soil aggregation seemed to capture aspects of actual vulnerability that are not visible through the erodibility estimate. Considering the relevance and extension of agrosilvopastoral ecosystems partly left to natural colonization, further studies on litter and humus protective action might improve the understanding of the relationship among erosion, erodibility, and structure.