New modelling tools are required to accelerate the decarbonisation of the building sector. Urban building energy modelling (UBEM) has recently emerged as an attractive paradigm for analysing building energy performance at district and urban scales. The balance between the fidelity and accuracy of created UBEMs is known to be the cornerstone of the model’s applicability. This study aimed to analyse the impact of traditionally implicit modeller choices that can greatly affect the overall UBEM performance, namely, (1) the level of detail (LoD) of the buildings’ geometry; (2) thermal zoning; and (3) the surrounding shadowing environment. The analysis was conducted for two urban areas in Stockholm (Sweden) using MUBES—the newly developed UBEM. It is a bottom-up physics-based open-source tool based on Python and EnergyPlus, allowing for calibration and co-simulation. At the building scale, significant impact was detected for all three factors. At the district scale, smaller effects (<2%) were observed for the level of detail and thermal zoning. However, up to 10% difference may be due to the surrounding shadowing environment, so it is recommended that this is considered when using UBEMs even for district scale analyses. Hence, assumptions embedded in UBEMs and the scale of analysis make a difference.