Deep reinforcement learning enables robots to learn collaborative tasks with humans. However, off-policy methods suffer from primacy bias that causes agents to overfit to early experiences. We investigate the impact of primacy bias on team performance during a real world human-robot co-learning task, where twenty novice human participants collaborated with a Soft Actor-Critic agent to move a UR3 cobot. Analysis of how initial interactions shape subsequent learning dynamics demonstrates that the quality of the initial data distribution matters. While successful early experiences allow teams to overcome learning barriers, poor interactions cause the agent to converge toward suboptimal behaviors that prevent recovery, even as human skills improve.