Most of the reinforced concrete buildings in Nepal are low-rise construction, as this type of construction is the most dominant structural form adopted to construct residential buildings in urban and semi-urban neighborhoods throughout the country. The low-rise residential constructions generally follow the guidelines recommended by the Nepal Building Code, especially the mandatory rules of thumb. Although low-rise buildings have brick infills and are randomly constructed, infill walls and soil–structure interaction effects are generally neglected in the design and assessment of such structures. To this end, bare frame models that are used to represent such structures are questionable, especially when seismic vulnerability analysis is concerned. To fulfil this gap, we performed seismic vulnerability analysis of low-rise residential RC buildings considering infill walls and soil–structure interaction effects. Considering four analysis cases, we outline comparative seismic vulnerability for various analysis cases in terms of fragility functions. The sum of observations highlights that the effects of infills, and soil–structure interaction are damage state sensitive for low-rise RC buildings. Meanwhile, the design considerations will be significantly affected since some performance parameters are more sensitive than the overall fragility. We also observed that the analytical fragility models fundamentally overestimate the actual seismic fragility in the case of low-rise RC buildings.