In the context of 3D geometric reconstruction from sequential inverse synthetic aperture radar (ISAR) images, the accurate scatterer trajectory association is a critical step. Aiming at the above problem, an enhanced scatterer trajectory association method is proposed by designing a modified label Gaussian mixture probability hypothesis density (ML-GM-PHD) filtering algorithm. The algorithm commences by constructing a general motion model for scatterers across sequential ISAR images, followed by an in-depth analysis of their motion characteristics. Subsequently, the actual projected positions and measurements of the scattering centers of the observed target are treated as random finite sets, which allows us to reformulate the scatterer trajectory association into a maximum a posteriori (MAP) estimation problem. After that, a ML-GM-PHD filtering algorithm is proposed to realize the scatterer trajectory association. Furthermore, the proposed method is applied to ISAR images in both the forward and reverse directions, enabling the fusion of trajectories from opposing directions to bolster the completeness of the scatterer trajectories. Finally, the factorization method is performed on the scatterer trajectory matrix to implement the 3D geometry reconstruction of the scattering centers in the observed target. Experimental results based on random points and electromagnetic data verify the effectiveness and performance priority of the proposed algorithm.