Accurately assessing rooftop photovoltaic (PV) potential requires precise identification of rooftop areas and availability. Current deep learning approaches using aerial imagery are faced with two challenges: inconsistent rooftop appearances caused by varying solar azimuths tend to mislead rooftop orientation extraction, and the existence of ancillary rooftop facilities often results in overestimation of solar potential. To tackle these challenges, a novel framework is proposed, with three components: automated extraction of rooftop areas and orientations, appearance-based estimation of rooftop availability coefficients, and PV potential calculation via a multi-orientation quantitative integration strategy. The segmentation network identifies geometric boundaries of rooftops and categorizes pitched roof segments into orientation-specific categories. High-level features of rooftop segments are then extracted from deep networks and clustered to compute availability coefficients at segment-level. Finally, the integration strategy leverages the symmetry assumption of sloped rooftops to mitigate classification errors and improve robustness in solar potential computation. Our framework is trained on the RID dataset with different category definition schemes, and estimation results are compared with solar radiation flux provided by NASA POWER. The overall relative error is less than 1%, which demonstrates the effectiveness of our framework.