Prior research shows that the GenderMag method can help identify and address usability barriers that are more likely to affect women software users than men. However, the evidence for the effectiveness of GenderMag is limited to small lab studies. In this case study, by combining self-reported gender data from tens of thousands of users of an internal code review tool with software logs data gathered over a five-year period, we quantitatively show that GenderMag helped a team at Google (a) correctly identify discoverability as a usability barrier more likely to affect women than men, and (b) increase discoverability by 2.4x while also achieving gender parity. That is, compared to men using the original code review tool, women and men using the system redesigned with GenderMag were both 2.4x more likely to discover the "Suggest Edit" feature at any given time. Thus, this paper contributes the first large-scale evidence of the effectiveness of GenderMag in the field.
Trust is central to how developers engage with AI. In this article, we discuss what we learned from developers about their level of trust in AI powered developer tooling, and how we translated those findings into product design recommendations to support customization, and the challenges we encountered along the way.
In past installments of our column, we’ve covered ideas about what developers want from AI and how our colleagues have sought to understand and design AI-powered developer tools that humans will trust, adopt, and use to become more productive. We invited some of those same colleagues to contribute their work again in this issue because we’re excited about how AI model quality, tool design, and human behavior come together in this work. Much current research focuses on quantifying the productivity impacts of AI for developers, but this article presents a comprehensive and practical example of the challenge inherent in designing and deploying AI in this manner. We hope that the combination of our prior articles and this one will help others to effectively guide human-centered, AI-driven developer productivity improvements. —Ciera Jaspan and Collin Green