The yellow fever (YF) vaccine is one of the safest and most effective vaccines currently available. Still, its administration in people living with HIV (PLWH) is limited due to safety concerns and a lack of consensus regarding decreased immunogenicity and long-lasting protection for this population. The mechanisms associated with impaired YF vaccine immunogenicity in PLWH are not fully understood, but the general immune deregulation during HIV infection may play an important role. To assess if HIV infection impacts YF vaccine immunogenicity and if markers of immune deregulation could predict lower immunogenicity, we evaluated the association of YF neutralization antibody (NAb) titers with the pre-vaccination frequency of activated and exhausted T cells, levels of pro-inflammatory cytokines, and frequency of T cells, B cells, and monocyte subsets in PLWH and HIV-negative controls. We observed impaired YF vaccine immunogenicity in PLWH with lower titers of YF-NAbs 30 days after vaccination, mainly in individuals with CD4 count <350 cells/mm³. At the baseline, those individuals were characterized by having a higher frequency of activated and exhausted T cells and tissue-like memory B cells. Elevated levels of those markers were also observed in individuals with CD4 count between 500 and 350 cells/mm³. We observed a negative correlation between the pre-vaccination level of CD8⁺ T cell exhaustion and CD4⁺ T cell activation with YF-NAb titers at D365 and the pre-vaccination level of IP-10 with YF-NAb titers at D30 and D365. Our results emphasize the impact of immune activation, exhaustion, and inflammation in YF vaccine immunogenicity in PLWH.