Although displacement measurement is essential for many civil infrastructure applications, the precise estimation of structural displacement remains a challenge. In this study, a structural displacement estimation technique was developed by fusing asynchronous acceleration and computer vision measurements using a Kalman filter. First, the scale factor, which converts translation from vision measurements (in pixel units) into displacement (in length units), is automatically computed using a natural target (i.e., without any artificial target or any prior knowledge of the target size). Second, an improved feature matching algorithm was developed to better trace the natural target within the computer vision. Third, an adaptive multirate Kalman filter was formulated such that asynchronous computer vision and acceleration measurements with different sampling rates could be seamlessly combined to improve displacement estimation. The feasibility and effectiveness of the proposed displacement estimation technique were validated by performing shaking table, four-story building model, and steel box girder pedestrian bridge tests. In all tests, the proposed technique was able to accurately estimate displacements with root mean square errors of less than 3 mm.