Gender is an important biological determinant of vulnerability to psychosocial stress. We used perfusion based functional magnetic resonance imaging (fMRI) to measure cerebral blood flow (CBF) responses to mild to moderate stress in 32 healthy people (16 males and 16 females). Psychological stress was elicited using mental arithmetic tasks under varying pressure. Stress in men was associated with CBF increase in the right prefrontal cortex (RPFC) and CBF reduction in the left orbitofrontal cortex (LOrF), a robust response that persisted beyond the stress task period. In contrast, stress in women primarily activated the limbic system, including the ventral striatum, putamen, insula and cingulate cortex. The asymmetric prefrontal activity in males was associated with a physiological index of stress responses—salivary cortisol, whereas the female limbic activation showed a lower degree of correlations with cortisol. Conjunction analyses indicated only a small degree of overlap between the stress networks in men and women at the threshold level of P < 0.01. Increased overlap of stress networks between the two genders was revealed when the threshold for conjunction analyses was relaxed to P < 0.05. Further, machine classification was used to differentiate the central stress responses between the two genders with over 94% accuracy. Our study may represent an initial step in uncovering the neurobiological basis underlying the contrasting health consequences of psychosocial stress in men and women.