Bridge health assessment has been a challenging problem due to great evaluation errors caused by heterogeneous and different dimensional bridge factors. Most approaches employ monitoring data of one bridge factor to simplify the evaluation problem resulting in poor assessment performance. To address this issue, this paper proposes an end-to-end framework to evaluate the health of bridges by exploring objective features and correlations of multiple monitoring factors. This model aims to learn representative features from raw monitoring data of bridge factors (i.e., strain, temperature, traffic flow, and heavy vehicle number), and classify the comprehensive features into different health degrees in a single framework. Specifically, in terms of characteristics of bridge factors (e.g., time sequence and heterogeneity), a hierarchical learning structure with multiple convolutional, pooling, and dense layers is presented to learn the representations of the whole bridge monitoring data. The structure contributes to capturing rich information from different observed factors and improving feature learning ability through designing particular neural networks for each bridge factor in accordance with their corresponding data structure. Additionally, a classification scheme with multiple fully connection layers and support vector machine in the framework is designed to achieve higher evaluation accuracy. Experiments on real-world monitoring data of a specific bridge validate the superiority of the proposed model.