A variety of ethical concerns about artificial intelligence (AI) implementation in healthcare have emerged as AI becomes increasingly applicable and technologically advanced. The last decade has witnessed significant endeavors in striking a balance between ethical considerations and health transformation led by AI. Despite a growing interest in AI ethics, implementing AI-related technologies and initiatives responsibly in healthcare settings remains a challenge. In response to this topical challenge, we reviewed 253 articles pertaining to AI ethics in healthcare published between 2000 and 2020, summarizing the coherent themes of responsible AI initiatives. A preferred reporting items for systematic review and meta-analysis (PRISMA) approach was employed to screen and select articles, and a hermeneutic approach was adopted to conduct systematic literature review. By synthesizing relevant knowledge from AI governance and ethics, we propose a responsible AI initiative framework that encompasses five core themes for AI solution developers, healthcare professionals, and policy makers. These themes are summarized in the acronym SHIFT: Sustainability, Human centeredness, Inclusiveness, Fairness, and Transparency. In addition, we unravel the key issues and challenges concerning responsible AI use in healthcare, and outline avenues for future research.