Stress has been identified as a main factor involved in the cognitive changes that occur during the aging process. This study investigated sex differences in the relationship between the magnitude of the acute stress-induced salivary cortisol response and memory performance among middle-aged people. To this end, 16 men and 16 women (aged 54–72 years) were exposed to the Trier Social Stress Test and a control condition in a crossover design. Afterwards their memory performance was measured using a standardized memory test (Rey's Auditory Verbal Learning Test). Only among women, there was an acute impact of stress on memory performance and a significant relationship between a higher cortisol response to the stressor and poorer memory performance in both the stress and control conditions. Additionally, a poorer memory performance was related to earlier timing of sexual maturation (age at menarche), which was also marginally related to higher cortisol reactivity to stress. These results confirm that sex is a critical factor in the relationship between cortisol and poor memory performance. Furthermore, the findings emphasize a strong link between the individual cortisol response to stress and memory functioning among postmenopausal women.