In this article, an adjustable autonomy framework is proposed for the human–robot collaboration (HRC) in which a robot uses a reinforcement learning mechanism guided by a human operator’s rewards in an initially unknown workspace. Within the proposed framework, the autonomy level of the robot is automatically adjusted in an HRC setting that is represented by a Markov decision process model. When the robot reaches higher performance levels, it can operate more autonomously in the sense that it needs less human operator intervention. A novel $Q$ -learning mechanism with an integrated $\epsilon$ -greedy approach is implemented for robot learning in order to capture the correct actions and robot’s mistakes as a basis for adjusting the robot’s autonomy level. The proposed HRC framework can adapt to changes in the workspace as well as changes in the human operator reward (scaling and shifting) mechanism, and can always adjust the autonomy level. The autonomy level of the robot is automatically lowered when the workspace changes to allow the robot to explore new actions in order to adapt to the new workspace. In addition, the human operator has the ability to reset/lower the autonomy level of the robot to enforce the robot to relearn the workspace if its performance is not satisfactory for the human operator. The developed algorithm is applied to a realistic HRC setting involving a humanoid robot, named Baxter. The experimental results are analyzed to assess the effectiveness of the proposed adjustable autonomy framework for different cases: for the case when the workspace does not change, then for the case when the robot autonomy level is reset/lowered by a human operator, and for the case when the workspace is changed by the introduction of new objects. The results confirm the capability of the developed framework to successfully adjust the autonomy level in response to changes in the human operator’s commands or the workspace.