Autonomous vehicles (AVs) are increasingly common, although there remain a number of limitations that need to be addressed in order for their deployment to be more widespread. For example, to mitigate the failure of self-driving functions in AVs, introducing the remote control capability (which allows a human driver to operate the vehicle remotely in certain circumferences) is one of several countermeasures proposed. However, the remote control capability breaks the isolation of onboard driving systems and can be potentially exploited by malicious actors to take over control of the AVs; thus, risking the safety of the passengers and pedestrians (e.g., AVs are remotely taken over by terrorist groups to carry out coordinated attacks in places of mass gatherings). Therefore, security is a key, mandatory feature in the design of AVs. In this paper, we propose a cloud-centric three-factor authentication and key agreement protocol (CT-AKA) integrating passwords, biometrics and smart cards to ensure secure access to both cloud and AVs. Three typical biometric encryption approaches, including fuzzy vault, fuzzy commitment, and fuzzy extractor, are unified to achieve three-factor authentication without leaking the biometric privacy of users. Moreover, two session keys are negotiated in our protocol, namely: one between the user and AV to support secure remote control of the AV, and the other is negotiated between the mobile device and the cloud to introduce resilience to the compromise of ephemeral security parameters to ensure cloud data access security with a high security guarantee. Finally, we formally verify the security properties and evaluate the efficiency of CT-AKA, whose findings demonstrate that the protocol achieves high security strength with reasonable computation and communication costs.