Michelle Y. Chen

Michelle Y. Chen

Auto-generated profile, based on publicly available information

Michelle Y. Chen

Paragon Testing Enterprises, Vancouver, British Columbia, Canada

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
18 October 2022 in Multivariate Behavioral Research
Verified
Journal Article
Closed access
YLYan Liu
Yan Liu
ABAudrey Béliveau
Audrey Béliveau
YWYaguang Wei
Yaguang Wei
MCMichelle Y. Chen
Michelle Y. Chen
RRRosalynn Record-Lemon
Rosalynn Record-Lemon
GCGuanyu Chen
Guanyu Chen
Network meta-analysis is an extension of standard meta-analysis. It allows researchers to build a network of evidence to compare multiple interventions that may have not been compared directly in existing publications. With a Bayesian approach, network meta-analysis can be used to obtain a posterior probability distribution of all the relative treatment effects, which allows for the estimation of relative treatment effects to quantify the uncertainty of parameter estimates, and to rank all the treatments in the network. Ranking treatments using both direct and indirect evidence can provide guidance to policy makers and clinicians for making decisions. The purpose of this paper is to introduce fundamental concepts of Bayesian network meta-analysis (BNMA) to researchers in psychology and social sciences. We discuss several essential concepts of BNMA, including the assumptions of homogeneity and consistency, the fixed and random effects models, prior specification, and model fit evaluation strategies, while pointing out some issues and areas where researchers should use caution in the application of BNMA. Additionally, using an automated R package, we provide a step-by-step demonstration on how to conduct and report the findings of BNMA with a real dataset of psychological interventions extracted from PubMed.

4 October 2019 in Educational and Psychological Measurement
Verified
Journal Article
Closed access
PDF file
MCMichelle Y. Chen
Michelle Y. Chen
YLYan Liu
Yan Liu
BZBruno D. Zumbo
Bruno D. Zumbo
This study introduces a novel differential item functioning (DIF) method based on propensity score matching that tackles two challenges in analyzing performance assessment data, that is, continuous task scores and lack of a reliable internal variable as a proxy for ability or aptitude. The proposed DIF method consists of two main stages. First, propensity score matching is used to eliminate preexisting group differences before the test, ideally creating equivalent groups as in a randomized experimental study. Then, linear mixed effects models are adopted to perform DIF analysis based on the matched data set. We demonstrate this propensity DIF method using a high-stakes functional English language proficiency test. DIF due to education was investigated in the writing component, which consists of two continuously scored performance-based tasks. Although the proposed method is demonstrated in the context of language testing, it can be applied to other types of performance assessments.

26 October 2020 in International Journal of Lifelong Education
Verified
Journal Article
Closed access
APAshley Pullman
Ashley Pullman
MCMichelle Y. Chen
Michelle Y. Chen
According to practice engagement theory within the field of adult education, everyday workplace activities offer an opportunity for skill development among youth and young adults; yet, prior research highlights the prevalence of routine and low-skill work within entry-level jobs. Utilising data from the Programme for the International Assessment of Adult Competencies (PIAAC), the objective of the following study is to demonstrate how workplace literacy, numeracy, and problem-solving activities differ among people aged 16 to 34 in Canada. Latent class analysis provides a typology of literacy, numeracy, and problem-solving activities at work and generates insight into how socio-demographic and employment factors are associated with each group. The findings highlight that employment cannot be uniformly characterised as low-skill. Rather, there are unique employment groups with differing literacy, numeracy, and problem-solving activity characteristics. Nevertheless, socio-demographic factors do matter for gaining access to employment that involves daily skill-based activities. While age, parental education, immigration background, and education level are significantly associated with the latent classes, economic and employment characteristics reduce the magnitude of these associations. Together, the results suggest that access to skill-based activities at work are more or less accessible for certain groups and connect to other aspects of early-career job quality and occupational segregation.

16 October 2020 in Canadian Journal of Applied Linguistics / Revue Canadienne de Linguistique Appliquée
Verified
Journal Article
Open access
PDF file
MCMichelle Chen
Michelle Chen
JFJennifer J. Flasko
Jennifer J. Flasko
Seeking evidence to support content validity is essential to test validation. This is especially the case in contexts where test scores are interpreted in relation to external proficiency standards and where new test content is constantly being produced to meet test administration and security demands. In this paper, we describe a modified scale- anchoring approach to assessing the alignment between the Canadian English Language Proficiency Index Program (CELPIP) test and the Canadian Language Benchmarks (CLB), the proficiency framework to which the test scores are linked. We discuss how proficiency frameworks such as the CLB can be used to support the content validation of large-scale standardized tests through an evaluation of the alignment between the test content and the performance standards. By sharing both the positive implications and challenges of working with the CLB in high-stakes language test validation, we hope to help raise the profile of this national language framework among scholars and practitioners.

5 July 2019 in Journal of Psychoeducational Assessment
Verified
Journal Article
Closed access
PDF file
MCMichelle Y. Chen
Michelle Y. Chen
AWAmery D. Wu
Amery D. Wu
YLYan Liu
Yan Liu
Answering the call for response process–based validation, this study shows how researchers can evaluate validity evidence of test scores based on the mental “processes” test-takers use, rather than based on correlations with other “outcome” measures. The proposed methods for process-based validation studies are demonstrated using a sample of 189 adults who took two listening comprehension tasks. Immediately after completing each task, the test-takers filled out a 10-item survey to reflect on the mental processes involved in reaching their answers. These 10 process variables attempted to capture five desired and five undesired response processes in answering multiple-choice listening comprehension questions. We investigated the relationships between these process variables and the binary outcome of item score (correct vs. incorrect) using mixed-effects logistic regression models, and showed how the results could provide validity evidence (or lack thereof). By doing so, we offer an alternative approach to study response process and test performance and encourage more process-based validation studies.

3 June 2020 in Frontiers in Psychology
Verified
Journal Article
Open access
PDF file
ZLZhi Li
Zhi Li
MCMichelle Y. Chen
Michelle Y. Chen
JBJayanti Banerjee
Jayanti Banerjee
Addressing differential item functioning (DIF) provides validity evidence to support the interpretation of test scores across groups. Conventional DIF methods flag DIF items statistically, but often fail to consolidate a substantive interpretation. The lack of interpretability of DIF results is particularly pronounced in writing assessment where the matching of test takers’ proficiency levels often relies on external variables and the reported DIF effect is frequently small in magnitude. Using responses to a prompt that showed small gender DIF favoring female test takers, we demonstrate a corpus-based approach that helps address DIF interpretation. To provide linguistic insights into the possible sources of the small DIF effect, this study compared a gender-balanced corpus of 826 writing samples matched by test takers’ performance on the reading and listening components of the test. Four groups of linguistic features that correspond to the rating dimensions, and thus partially represent the writing construct were analyzed. They include (1) sentiment and social cognition, (2) cohesion, (3) syntactic features, and (4) lexical features. After initial screening, 123 linguistic features, all of which were correlated with the writing scores, were retained for gender comparison. Among these selected features, female test takers’ writing samples scored higher on six of them with small effect sizes in the categories of cohesion and syntactic features. Three of the six features were positively correlated with higher writing scores, while the other three were negative. These results are largely consistent with previous findings of gender differences in language use. Additionally, the small differences in the language features of the writing samples (in terms of the small number of features that differ between genders and the small effect size of the observed differences) are consistent with the previous DIF results, both suggesting that the effect of gender differences on the writing scores is likely to be very small. In sum, the corpus-based findings provide linguistic insights into the gender-related language differences and their potential consequences in a testing context. These findings are meaningful for furthering our understanding of the small gender DIF effect identified through statistical analysis, which lends support to the validity of writing scores.