Some asteroids flying close to Earth may pose a threat of impact. Among them, the structural and dynamical characteristics of rubble-pile asteroids can be changed because of the tidal force of the Earth in this process. This can provide key information for predicting the dynamical evolution of potentially hazardous asteroids. In this study, the long-term evolution of the coupling orbit–attitude–structure of these small bodies is presented numerically based on the integration of two models. One is the 3D discrete element method, which models the structure and irregular shape of the rubble-pile asteroid. The other is the dynamical model of the circular restricted three-body problem (CRTBP). This provides a more precise dynamical environment of the asteroid orbital deflection, morphological modification, and attitude angles analysis compared to the frequently adopted two-body problem. Parametric studies on the asteroid evolution were performed focusing on its flyby distance and the bulk porosity. Numerical results indicate that the Earth flyby can form different patterns of modification of asteroids, where the rubble-pile structure can be destructed by considering the bulk porosity. The asteroid orbital deflection and attitude variational trends are also summarized based on the simulations of multi-orbital revolutions.