Previous research has suggested that there are two modular processes of perspective selection and calculation in level 1 perspective taking. Evidence further showed that verbal working memory is associated with both processes in adults. However, research has not tested whether verbal working memory may be associated with working memory in children. Moreover, since perspective taking is associated with spatial working memory, it is necessary to investigate whether spatial working memory links to both processes. By recruiting 9-year-old children and college students in the single-task paradigm and the dual-task paradigm, we conducted two experiments to answer these questions. Results in experiment 1 suggested that verbal working memory correlated with adults’ perspective calculation, but with both processes in children’s perspective taking. Results in experiment 2 showed that spatial working memory is associated with adults’ perspective calculation and children’s perspective selection. These findings suggest that different components of working memory play distinct roles in the processing of perspective taking, which is moderated by age.