Rural residents not only produce rural garbage and participate in its disposal, but are also beneficiaries of a beautiful rural environment. The garbage exchange supermarket (where garbage is exchanged for goods) is a garbage disposal method that is employed in some villages in China. It is of great significance for the improvement of rural living environment and rural residents’ awareness of environmental protection. Thus, it is necessary to explore rural residents’ perceptions and behavior regarding garbage exchange supermarkets. Based on planned behavior theory and social exchange theory, this paper develops a model of rural residents’ perceptions, attitudes, and environmentally responsible behaviors regarding garbage exchange supermarkets. Then, using Huangshan City, China, as a case study, three villages, located in the upper, middle, and lower reaches of the Xin’an River were selected. Using a stratified sampling method, 324 questionnaires were obtained from residents. The developed model was verified by the method of structural equation modeling. Findings are as follows: (1) On the whole, residents have a strong and positive perception of the benefits of garbage exchange supermarkets, with an emphasis on its environmental advantages. (2) Regarding the cost dimension in perception, the focus is spent queuing for exchange on the time and sorting garbage at home. In general, residents are still willing to spend this time going to the supermarket to exchange. (3) Environmentally responsible behavior is divided into two dimensions: compliance and promotion-type environmentally responsible behavior—the former is more apparent among rural residents. (4) Residents’ perceptions of benefits positively affect their attitudes and satisfaction towards garbage exchange supermarkets. Cost perception has no significant effect on residents’ attitudes but has a negative correlation with satisfaction, satisfaction and attitude have positive correlations with environmentally responsible behavior, and satisfaction also positively affects residents’ attitudes.