Rotational inertia damper, a novel damper, possessing the advantage of displacement amplification, has been employed in outrigger system for seismic mitigation. The equivalent analysis model composed by a uniform cantilever beam and an equivalent spring was proposed to simulate the rotational inertia damper outrigger system, by which the corresponding dynamic characteristic equation was derived based on numerical assembly technique. To gain the response of the damped system, finite element method and state space method have been utilized. Finally, the results show that the pseudo-undamped natural frequency ratios and system modal damping ratios are significantly influenced by stiffness parameter of the exterior column, while the mass parameter of the rotational inertia damper has little effect on them. The optimal damping ratio can be acquired for one mode, but it may be worse for the other mode in the same position equipping rotational inertia damper. Furthermore, numerical simulation results for the typical earthquake records have verified that the rotational inertia damper outrigger has excellent control performance in displacement as well as acceleration. A good agreement between damping force and equivalent force also suggests that the damping force of rotational inertia damper is predominant and the inertial force has no significant effect on the structure.