As use and availability of mobile health apps have increased, so too has the need for a thorough, accessible framework for app evaluation. The American Psychiatric Association’s app evaluation model has emerged as a way to critically assess an app by considering accessibility, privacy and security, clinical foundation, engagement, and interoperability; however, there is no centralized database where users can view how various health apps perform when assessed via the APA model. In this perspective, we propose and outline our effort to translate the APA’s model for the evaluation of health apps into a set of objective metrics that can be published online, making the framework actionable and accessible to a broad audience. The questions from the APA model were operationalized into 105 objective questions that are either binary or numeric. These questions serve as the foundation of an online database, where app evaluation consists of answering these 105 questions and can be crowdsourced. While the database has yet to be published and crowdsourced, initial internal testing demonstrated excellent interrater reliability. The database proposed here introduces a public and interactive approach to data collection that is guided by the APA model. The published product enables users to sort through the many mobile health apps and filter them according to individual preferences and priorities, making the ever-growing health app market more navigable.
Objectives Despite an estimated 300 000 mobile health apps on the market, there remains no consensus around helping patients and clinicians select safe and effective apps. In 2018, our team drew on existing evaluation frameworks to identify salient categories and create a new framework endorsed by the American Psychiatric Association (APA). We have since created a more expanded and operational framework Mhealth Index and Navigation Database (MIND) that aligns with the APA categories but includes objective and auditable questions (105). We sought to survey the existing space, conducting a review of all mobile health app evaluation frameworks published since 2018, and demonstrate the comprehensiveness of this new model by comparing it to existing and emerging frameworks. Design We conducted a scoping review of mobile health app evaluation frameworks. Data sources References were identified through searches of PubMed, EMBASE and PsychINFO with publication date between January 2018 and October 2020. Eligibility criteria Papers were selected for inclusion if they meet the predetermined eligibility criteria—presenting an evaluation framework for mobile health apps with patient, clinician or end user-facing questions. Data extraction and synthesis Two reviewers screened the literature separately and applied the inclusion criteria. The data extracted from the papers included: author and dates of publication, source affiliation, country of origin, name of framework, study design, description of framework, intended audience/user and framework scoring system. We then compiled a collection of more than 1701 questions across 79 frameworks. We compared and grouped these questions using the MIND framework as a reference. We sought to identify the most common domains of evaluation while assessing the comprehensiveness and flexibility—as well as any potential gaps—of MIND. Results New app evaluation frameworks continue to emerge and expand. Since our 2019 review of the app evaluation framework space, more frameworks include questions around privacy (43) and clinical foundation (57), reflecting an increased focus on issues of app security and evidence base. The majority of mapped frameworks overlapped with at least half of the MIND categories. The results of this search have informed a database (apps.digitalpsych.org) that users can access today. Conclusion As the number of app evaluation frameworks continues to rise, it is becoming difficult for users to select both an appropriate evaluation tool and to find an appropriate health app. This review provides a comparison of what different app evaluation frameworks are offering, where the field is converging and new priorities for improving clinical guidance.
Background While there are numerous mental health apps on the market today, less is known about their safety and quality. This study aims to offer a longitudinal perspective on the nature of high visibility apps for common mental health and physical health conditions. Methods In July 2019, we selected the 10 top search-returned apps in the Apple App Store and Android Google Play Store using six keyword terms: depression, anxiety, schizophrenia, addiction, high blood pressure and diabetes. Each app was downloaded by two authors and reviewed by a clinician, and the app was coded for features, functionality, claims, app store properties, and other properties. Results Compared with 1 year prior, there were few statistically significant changes in app privacy policies, evidence and features. However, there was a high rate of turnover with only 34 (57%) of the apps from the Apple’s App Store and 28 (47%) from the Google Play Store remaining in the 2019 top 10 search compared with the 2018 search. Discussion Although there was a high turnover of top search-returned apps between 2018 and 2019, we found that there were few significant changes in features, privacy, medical claims and other properties. This suggests that, although the highly visible and available apps are changing, there were no significant improvements in app quality or safety.
27 August 2020 in Journal of Medical Internet Research
Background: Despite the emergence of app evaluation tools, there remains no well-defined process receptive to diverse local needs, rigorous standards, and current content. The need for such a process to assist in the implementation of app evaluation across all medical fields is evident. Such a process has the potential to increase stakeholder engagement and catalyze interest and engagement with present-day app evaluation models. Objective: This study aimed to develop and pilot test the Technology Evaluation and Assessment Criteria for Health apps (TEACH-apps). Methods: Tailoring a well-known implementation framework, Replicating Effective Programs, we present a new process to approach the challenges faced in implementing app evaluation tools today. As a culmination of our experience implementing this process and feedback from stakeholders, we present the four-part process to aid the implementation of mobile health technology. This paper outlines the theory, evidence, and initial versions of the process. Results: The TEACH-apps process is designed to be broadly usable and widely applicable across all fields of health. The process comprises four parts: (1) preconditions (eg, gathering apps and considering local needs), (2) preimplementation (eg, customizing criteria and offering digital skills training), (3) implementation (eg, evaluating apps and creating educational handouts), and (4) maintenance and evolution (eg, repeating the process every 90 days and updating content). TEACH-apps has been tested internally at our hospital, and there is growing interest in partnering health care facilities to test the system at their sites. Conclusions: This implementation framework introduces a process that equips stakeholders, clinicians, and users with the foundational tools to make informed decisions around app use and increase app evaluation engagement. The application of this process may lead to the selection of more culturally appropriate and clinically relevant tools in health care.
Smartphone applications related to coronavirus disease 2019 (COVID-19) continue to emerge and evolve, but despite a wide variety of different app functions, there has yet to be a comprehensive study of what the most prevalent publicly available apps provide, and there exists no standardized evaluation system for end users to determine the safety and efficacy of an app before they download it. Furthermore, limited oversight means that the rapidly growing space creates challenges for end users trying to find a relevant app. We adapted the M-Health Index and Navigation Database (MIND) from apps.digitalpsych.org that previously has been used to evaluate mental health applications to guide the assessment of COVID apps. Using this framework, we conducted a thorough analysis of the top-100 returned coronavirus apps on two separate dates a month apart to understand the clinical utility and features of COVID-19 apps and how these change in a short period of time. We ultimately identified a significant turnover rate, as well as privacy concerns around lack of privacy policies and disclosure of personal information. Our research offers insight into the current status of COVID-19 apps and provides a comprehensive and adaptable framework to help individuals assess the growing number of such digital tools in the wake of the pandemic.
Background Despite a recent proliferation in web-based and digital resources that are designed to assist users in finding appropriate mental health treatment and supportive services, it can be overwhelming, confusing, and difficult for an individual or family member to access and use an appropriate navigation tool. As digital resources are increasingly sought after, there is an urgent need for a clearer understanding of digital navigation tools in order to help link individuals with the tool that is best suited to their needs. Objective The objective of this study was to determine the needs of individuals seeking mental health treatment and supportive services and to quantify their experiences and satisfaction with available digital navigation tools. Methods A survey was offered via an email newsletter and social media posting throughout the extended membership of the National Alliance on Mental Illness, which includes both individuals with a mental health condition and their family members and support networks. A 13-item anonymous survey, which consisted of multiple-choice and open response options, was developed to measure participants’ past use of and experiences with web-based, mobile, and phone-based navigation tools. The survey was available from April 9 through May 21, 2020. Results A total of 478 respondents completed the survey; the majority of respondents were female (397/478, 83.1%) and aged ≥35 years (411/478, 86%). Younger respondents were more likely to report seeking mental health services for themselves, while older respondents were more likely to be searching for such services on behalf of a family member. The majority of respondents seeking such services on behalf of a family member (113/194, 58.2%) required a combination of mental health treatment and supportive services. Furthermore, two-thirds of respondents (322/478, 67.4%) used a navigation tool to find treatment or services. The majority of respondents who provided feedback about their experiences with navigation tools (224/280, 80%) reported difficulties, with data availability and accuracy being the most commonly reported issues. Conclusions The survey results suggest that issues with data availability and accuracy in available navigation tools remain a major barrier for locating timely and appropriate mental health treatment and supportive services within the population of individuals seeking such services. Particularly for individuals seeking care on behalf of a family member, improving the accuracy of and users’ experiences with navigation tools could have a major impact on effectively connecting people to treatment and support services.