4 June 2009 in Applied Linguistics
In the rating scales of major international language tests, as well as in automated evaluation systems (e.g. e-rater), a positive relationship is often claimed between lexical diversity, holistic quality of written or spoken discourses, and language proficiency of candidates. This article reports a posteriori validation study that analysed a sample of the archived data of an international language test to examine empirically to what extent such relationships exist. It is also noted that previous studies on lexical diversity in the field of applied linguistics have focused exclusively on either written or spoken discourses, no study to date has compared lexical diversity of spoken and written discourses produced by the same participants. Therefore, the second aim of this article is to understand the differences in lexical diversity between writing and speaking task performances, and to what extent the topics of the writing prompts may affect lexical diversity of written discourses. Using D as a measure of lexical diversity (Malvern and Richards 1997, 2002; Malvern et al. 2004), it was found that D had a statistically significant and positive correlation with the overall quality ratings of both writing and speaking performances as well as the candidates’ general language proficiency. Nevertheless, the significant relationships were not borne out across the subgroups of the sample in terms of gender, first language background, purpose of taking the test and topics of the writing prompts. The different writing topics also had significant effects on lexical diversity—especially the topics that candidates were highly familiar with—even after controlling for writing ability and overall language proficiency. The lexical diversity of candidates’ writing and speaking performances were approximately at the same level; further, D was found to be a better predictor of speaking than writing performance. The implications of these findings are discussed with specific reference to the use of lexical diversity measures to inform language test validation and the development of lexical diversity parameters in automated evaluation systems.
2 April 2016 in Educational Research
Background: PISA results appear to have a large impact upon government policy. The phenomenon is growing, with more countries taking part in PISA testing and politicians pointing to PISA results as reasons for their reforms. Purpose: The aims of this research were to depict the policy reactions to PISA across a number of jurisdictions, to see whether they exhibited similar patterns and whether the same reforms were evident. Sources of evidence: We investigated policy and media reactions to the 2009 and 2012 PISA results in six cases: Canada, China (Shanghai), England, France, Norway and Switzerland. Cases were selected to contrast high-performing jurisdictions (Canada, China) with average performers (England, France, Norway and Switzerland). Countries that had already been well reported on in the literature were excluded (Finland, Germany). Design and methods: Policy documents, media reports and academic articles in English, French, Mandarin and Norwegian relating to each of the cases were critically evaluated. Results: A policy reaction of ‘scandalisation’ was evident in four of the six cases; a technique used to motivate change. Five of the six cases showed ‘standards-based reforms’ and two had reforms in line with the ‘ideal-governance’ model. However, these are categorisations: the actual reforms had significant differences across countries. There are chronological problems with the notion that PISA results were causal with regard to policy in some instances. Countries with similar PISA results responded with different policies, reflecting their differing cultural and historical education system trajectories. Conclusions: The connection between PISA results and policy is not always obvious. The supranational spell of PISA in policy is in the way that PISA results are used as a magic wand in political rhetoric, as though they conjure particular policy choices. This serves as a distraction from the ideological basis for reforms. The same PISA results could motivate a range of different policy solutions.
1 January 2009 in Assessing Writing
15 November 2008 in Educational Research
Background: The Effective lifelong learning inventory (ELLI) has been used as a diagnostic self assessment tool in schools and universities and other learning contexts as a means of raising an individual's awareness of their own learning dispositions and encouraging them to take responsibility for their own learning. The demand for the ELLI, in practice, has been significant, and since it is administered online, since 2004 a dataset of over 10,000 cases has accumulated. Purpose: This paper focuses on an exploration of the internal reliability, validity and stability of the seven scales of the ELLI. Some suggestions of its relevance to practice are made, though this is not the focus of the paper and is discussed elsewhere. Programme description: The inventory is available to formal learning organisations, and is administered by teachers who have been trained in its application, drawing on research and best practice through an evaluated programme. Sample: The total sample in this study is 10,496 individuals from 122 institutions and 413 classrooms. Design and methods: An exploratory factor analysis and scale reliability computations were undertaken and a ‘sensitivity’ study based on a subgroup of the original sample, collected in 2002 is reported. An Analysis of Variance study is repeated, comparing means between age ranges. Results: The study demonstrates again that the scales remain stable and continue to reach acceptable reliability levels in the new sample, in five age ranges and that the mean score on all positive learning dispositions reduces significantly until the 16–19 age range. Conclusions: The scales demonstrate a significant degree of stability, reliability and internal consistency over time. The constructs measured are thus useful for research, and institutional self-evaluation, and can be used, in practice, with some confidence. There remains a significant ongoing research agenda relating to the plasticity of the construct of learning dispositions and the ‘positioning’ of dispositions within an ecology of learning.
2 July 2016 in Language Assessment Quarterly
Integrated assessment tasks have been increasingly used in language tests, but the underlying constructs of integrated tasks remain elusive. This study aimed to improve understanding of the construct of integrated writing tasks in Chinese Language examinations in Hong Kong by looking at the language competences measured in the Listening-Reading-Writing Task and how they relate to the outcome of the Independent Listening Task. The performance of 226 native Chinese Secondary Five students on both tasks were subject to correlation analysis, joint factor analysis, and regression analysis. It was found that the students’ performance in the Independent Listening Task and the Listening-Reading-Writing Task was statistically significantly correlated, but the two tasks did not seem to have common factors as shown in the joint factor analysis. The indicators of elaboration, evaluation, and creation in the Independent Listening Task were significantly correlated with multiple indicators in the Listening-Reading-Writing Task, and evaluation and creation together explained 8.9% of the variance in the total score of the Listening-Reading-Writing Task. The findings support the framework (i.e., the “four pillars” of integrated writing competence) applied in public examinations in Hong Kong. They also imply that the two types of writing tasks are complementary in the assessment of Chinese Language competence.
1 October 2008 in Language Testing
The cognitive demands of summary writing are dependent upon the type of summary to be produced. This paper reports part of a larger study in which 157 Chinese undergraduates were asked to write summaries of extended English texts in both English and Chinese. It examines the differential effects of the use of the two languages on summarization as a measure of reading comprehension, drawing upon data from students' actual test performances as well as their perceptions of such differential effects, as evidenced in the post-summarization questionnaire and interviews. It was found that the use of the different languages had significant effects on both summarization processes and products. Students wrote significantly longer Chinese summaries but these were rated consistently of poorer quality than English summaries. However, Chinese summarization was found to be a better measure of students' English reading abilities. The implications of these findings are discussed with specific reference to summarization task design, in particular, how the use of the first and target languages may affect test takers' performance as well as rater behaviours and the potential of using first language in large-scale international language testing programmes incorporating integrated reading/writing tasks as a measure of reading comprehension.