2 June 2023 in Journal of Medical Internet Research
Background: The current methods of evaluating cognitive functioning typically rely on a single time point to assess and characterize an individual’s performance. However, cognitive functioning fluctuates within individuals over time in relation to environmental, psychological, and physiological contexts. This limits the generalizability and diagnostic utility of single time point assessments, particularly among individuals who may exhibit large variations in cognition depending on physiological or psychological context (eg, those with type 1 diabetes [T1D], who may have fluctuating glucose concentrations throughout the day). Objective: We aimed to report the reliability and validity of cognitive ecological momentary assessment (EMA) as a method for understanding between-person differences and capturing within-person variation in cognition over time in a community sample and sample of adults with T1D. Methods: Cognitive performance was measured 3 times a day for 15 days in the sample of adults with T1D (n=198, recruited through endocrinology clinics) and for 10 days in the community sample (n=128, recruited from TestMyBrain, a web-based citizen science platform) using ultrabrief cognitive tests developed for cognitive EMA. Our cognitive EMA platform allowed for remote, automated assessment in participants’ natural environments, enabling the measurement of within-person cognitive variation without the burden of repeated laboratory or clinic visits. This allowed us to evaluate reliability and validity in samples that differed in their expected degree of cognitive variability as well as the method of recruitment. Results: The results demonstrate excellent between-person reliability (ranging from 0.95 to 0.99) and construct validity of cognitive EMA in both the sample of adults with T1D and community sample. Within-person reliability in both samples (ranging from 0.20 to 0.80) was comparable with that observed in previous studies in healthy older adults. As expected, the full-length baseline and EMA versions of TestMyBrain tests correlated highly with one another and loaded together on the expected cognitive domains when using exploratory factor analysis. Interruptions had higher negative impacts on accuracy-based outcomes (β=−.34 to −.26; all P values <.001) than on reaction time–based outcomes (β=−.07 to −.02; P<.001 to P=.40). Conclusions: We demonstrated that ultrabrief mobile assessments are both reliable and valid across 2 very different clinic versus community samples, despite the conditions in which cognitive EMAs are administered, which are often associated with more noise and variability. The psychometric characteristics described here should be leveraged appropriately depending on the goals of the cognitive assessment (eg, diagnostic vs everyday functioning) and the population being studied.
5 January 2023 in JMIR Diabetes
Background: Individuals with type 1 diabetes represent a population with important vulnerabilities to dynamic physiological, behavioral, and psychological interactions, as well as cognitive processes. Ecological momentary assessment (EMA), a methodological approach used to study intraindividual variation over time, has only recently been used to deliver cognitive assessments in daily life, and many methodological questions remain. The Glycemic Variability and Fluctuations in Cognitive Status in Adults with Type 1 Diabetes (GluCog) study uses EMA to deliver cognitive and self-report measures while simultaneously collecting passive interstitial glucose in adults with type 1 diabetes. Objective: We aimed to report the results of an EMA optimization pilot and how these data were used to refine the study design of the GluCog study. An optimization pilot was designed to determine whether low-frequency EMA (3 EMAs per day) over more days or high-frequency EMA (6 EMAs per day) for fewer days would result in a better EMA completion rate and capture more hypoglycemia episodes. The secondary aim was to reduce the number of cognitive EMA tasks from 6 to 3. Methods: Baseline cognitive tasks and psychological questionnaires were completed by all the participants (N=20), followed by EMA delivery of brief cognitive and self-report measures for 15 days while wearing a blinded continuous glucose monitor. These data were coded for the presence of hypoglycemia (80% EMA completion. However, more hypoglycemia episodes were captured during the schedule with the 3 EMAs per day than during the schedule with 6 EMAs per day. Conclusions: The results from this EMA optimization pilot guided key design decisions regarding the EMA frequency and study duration for the main GluCog study. The present report responds to the urgent need for systematic and detailed information on EMA study designs, particularly those using cognitive assessments coupled with physiological measures. Given the complexity of EMA studies, choosing the right instruments and assessment schedules is an important aspect of study design and subsequent data interpretation.
13 November 2019 in PsyArXiv
The replication crisis has brought about an increased focus on improving the reproducibility of psychological research (Open Science Collaboration, 2015). Although some failed replications reflect false-positives in original research findings, many are likely the result of low statistical power, which can cause failed replications even when an effect is real, no questionable research practices are used, and an experiment’s methodology is repeated perfectly. The present paper describes a simulation method (bootstrap resampling) that can be used in combination with pilot data or synthetic data to produce highly powered experimental designs. Unlike other commonly used power analysis approaches (e.g., G*Power), bootstrap resampling can be adapted to any experimental design to account for various factors that influence statistical power, including sample size, number of trials per condition, and participant exclusion criteria. Ignoring some of these factors (e.g., by using G*Power) can overestimate the power of a study or replication, increasing the likelihood that your findings will not replicate. By demonstrating how these factors influence the consistency of experimental findings, this paper provides examples of how simulation can be used to improve statistical power and reproducibility. Further, we provide a MATLAB toolbox that can be used to implement these simulation-based methods on existing pilot data (https://harvard-visionlab.github.io/power-sim).
21 April 2022 in Assessment
Face recognition assessments that use images of celebrities require not only face recognition ability but also pop-culture knowledge and successful recall of identifying information. Here, we introduce a task designed to measure face recognition more specifically: the Famous Faces Doppelgangers Test (FFDT). Participants ( N = 57,407) identified 40 celebrities paired with lookalike doppelgangers, allowing face recognition ability to be assessed without requiring information recall. In addition, participants reported whether they were familiar with each celebrity, allowing poor face recognition ability to be differentiated from low pop-culture knowledge. FFDT performance was reliable (
r
xx
= .80), similar across participants of different racial and ethnic groups, and more highly correlated with memory for faces ( r = .50) and self-reported face recognition ability ( r = .48) than processing speed ability ( r = .10). Thus, the FFDT is a reliable, valid, and specific measure of the ability to identify familiar faces, making it a promising new tool for assessing face recognition ability.
2 July 2020 in Developmental Psychology
Multifocal attention is the ability to simultaneously attend to multiple objects, and is critical for typical functioning. Although adults are able to use multifocal attention, little is known about the development of this ability. In two experiments, we investigated multifocal attention in 6-8-year-old children and adults using a child-friendly, computerized multiple object tracking task designed to encourage the use of multifocal attention. We also investigated whether multifocal attention in children is deployed independently across left and right hemifields of vision, as in adults. Our results suggest that children's capacity for multifocal attention increases significantly across middle childhood. We also found evidence that at least one signature of hemifield-independent multifocal attention, the bilateral field advantage, can be observed in children.
27 August 2022 in OSF Preprints
Can an inclusive test of face cognition meet or exceed the psychometric properties of a prominent less inclusive test? Here, we norm and validate an updated version of the influential Reading the Mind in the Eyes Test (RMET), a clinically significant neuropsychiatric paradigm that has long been used to assess theory of mind and social cognition. Unlike the RMET, our Multiracial Reading the Mind in the Eyes Test (MRMET) incorporates racially inclusive stimuli, non-gendered answer choices, ground-truth referenced answers, and more accessible vocabulary. We show, via a series of large datasets, that the MRMET meets or exceeds RMET across major psychometric indices. Moreover, the reliable signal captured by the two tests is statistically indistinguishable, evidence for full interchangeability. We thus present the MRMET as a high-quality, inclusive, normed and validated alternative to the RMET, and as a case-in-point that inclusivity in psychometric tests of face cognition is an achievable aim. The MRMET test and our normative and validation data sets are openly available under a CC-BY-SA 4.0 license at osf.io/ahq6n.