This paper proposes the extended state observer (ESO)-based active disturbance rejection control (ADRC) for the speed tracking of an autonomous vehicle. Uncertainties, both in the vehicle plant and in the sensors, such as nonlinear uncertainties due to the powertrain dynamics, variations in rolling resistance and air resistance, are all estimated in real-time by an extended state observer (ESO). Furthermore, a simple vehicle longitudinal dynamics model, including a mean value engine model (MVEM), is implemented to obtain the parameters in ADRC and design a feedforward controller to enhance the controller’s performance. The proposed controller is validated through CarSim^®/Simulink^® simulations and road tests. The simulation validates the adaptiveness of the proposed controller against the well-tuned proportional integral derivative (PID) controller, and the speed tracking error of the proposed controller is within 1.26% in simulation. Simulation results also show that fuel consumption can be improved by 3.6% by changing the accelerator pedal depth and positive rate. Finally, the road tests are completed under four kinds of road conditions, and the maximum tracking error is smaller than 0.5 km/h.