Olfactory dysfunction (OD) is a common sequela of SARS-CoV-2 infection, but its prevalence and associated factors beyond 2 years remain unclear. In this multicenter observational study, we assessed 155 recovered COVID-19 patients approximately three years after infection and included 170 age-matched healthy controls as a reference group. Demographic, clinical, psychological, and sleep-related data were collected through structured interviews. Olfactory function at the 3-year assessment was objectively evaluated using Toyota–Takagi (T&T) olfactometry. Paired baseline-to-follow-up T&T data were available only for a small exploratory subgroup of nine patients and were analyzed descriptively. At follow-up, 7 of 155 recovered patients (4.5%) met our T&T-based definition of persistent quantitative olfactory dysfunction, all of whom were older than 50 years. Emotional and sleep disturbances were also common, with descriptive trends toward higher frequencies among women and older individuals. Exploratory analyses suggested that insomnia (AIS > 6; OR 5.35, 95% CI 1.07–26.60; p = 0.033) and anxiety (HAMA ≥ 7; OR 10.54, 95% CI 1.21–91.82; p = 0.04) were associated with persistent T&T-defined quantitative OD, although the small number of outcome events limited statistical precision. These findings indicate that a small proportion of COVID-19 survivors have persistent objective OD 3 years after infection and that persistent OD is associated with anxiety and insomnia.