This paper proposes a novel output-only structural damage indicator by incorporating the pole-based optimal subpattern assignment distance with autoregressive models to localize and relatively assess the severity of damages for sheared structures. Autoregressive models can model dynamic systems well, while their model poles can represent the state of the dynamic systems. Structural damage generally causes changes in the dynamic characteristics (especially the natural frequency, mode shapes and damping ratio) of structures. Since the poles of the autoregressive models can solve the modal parameters of the structure, the poles have a close relationship with the modal parameters so that the changes in the poles of its autoregressive model reflect structural damages. Therefore, we can identify the damage by tracking the shifts in the dynamic system poles. The optimal subpattern assignment distance, which is the performance evaluator in multi-target tracking algorithms to measure the metric between true and estimated tracks, enables the construction of damage sensitive indicator from system poles using the Hungarian algorithm. The proposed approach has been validated with a five-story shear-building using numerical simulations and experimental verifications, which are subjected to excitations of white noise, El Centro earthquake and sinusoidal wave with frequencies sweeping, respectively; the results indicate that this approach can localize and quantify structural damages effectively in an output-only and data-driven way.