Artificial intelligence (AI) and machine learning (ML) are increasingly broadly adopted in industry. However, based on well over a dozen case studies, we have learned that deploying industry-strength, production quality ML models in systems proves to be challenging. Companies experience challenges related to data quality, design methods and processes, performance of models as well as deployment and compliance. We learned that a new, structured engineering approach is required to construct and evolve systems that contain ML/DL components. In this chapter, the authors provide a conceptualization of the typical evolution patterns that companies experience when employing ML as well as an overview of the key problems experienced by the companies that they have studied. The main contribution of the chapter is a research agenda for AI engineering that provides an overview of the key engineering challenges surrounding ML solutions and an overview of open items that need to be addressed by the research community at large.
Companies need to know how much value their ideas deliver to customers. One of the most powerful ways to accurately measure this is by conducting online controlled experiments (OCEs). To run experiments, however, companies need to develop strong experimentation practices as well as align their organization and culture to experimentation. The main objective of this paper is to demonstrate how to run OCEs at large scale using the experience of companies that succeeded in scaling. Based on case study research at Microsoft, Booking.com, Skyscanner, and Intuit, we present our main contribution—The Experiment Growth Model. This four‐stage model addresses the seven critical aspects of experimentation and can help companies to transform their organizations into learning laboratories where new ideas can be tested with scientific accuracy. Ultimately, this should lead to better products and services.