The purpose of this paper is to investigate the effect of insufficient effort responding (IER) on construct validity of student evaluations of teaching (SET) in higher education. A total of 13,340 SET surveys collected by a major Jordanian university to assess teaching effectiveness were analyzed in this study. The detection method was used to detect IER, and the construct (factorial) validity was assessed using confirmatory factor analysis (CFA) and principal component analysis (PCA) before and after removing detected IER. The results of this study show that 2,160 SET surveys were flagged as insufficient effort responses out of 13,340 surveys. This figure represents 16.2 percent of the sample. Moreover, the results of CFA and PCA show that removing detected IER statistically enhanced the construct (factorial) validity of the SET survey. Since IER responses are often ignored by researchers and practitioners in industrial and organizational psychology (Liu et al., 2013), the results of this study strongly suggest that higher education administrations should give the necessary attention to IER responses, as SET results are used in making critical decisions The results of the current study recommend universities to carefully design online SET surveys, and provide the students with clear instructions in order to minimize students’ engagement in IER. Moreover, since SET results are used in making critical decisions, higher education administrations should give the necessary attention to IER by examining the IERs rate in their data sets and its consequences on the data quality. Reviewing the related literature shows that this is the first study that investigates the effect of IER on construct validity of SET in higher education using an IRT-based detection method.
27 January 2025 in Asian Education and Development Studies
Purpose: The objective of this study is to explore the structural relationships among context (C), input (I) and process (P) and product (P) (CIPP) components of teacher preparation programs based on students' perceptions.Design/methodology/approach: In this study, data were collected using a 17-item scale. The study sample consisted of 213 pre-service teachers enrolled in the Postgraduate Professional Diploma in Teaching (PPDT). Quantitative research methodology with multivariate structural equation modeling (SEM) was utilized to examine the two suggested models.Findings: The results of this study show that the CIPP model can be projected into preservice teachers' perceptions of the CIPP components. These preservice teachers' perceptions of preparation programs from the three components (CIP) can predict preservice teachers' perceptions of teachers’ preparation program products or outcomes (i.e. the fourth CIPP component). This result indicates that the relationships between the CIPP components and the pre-service teachers' perceptions of the Diploma in Teaching program are direct.Originality/value: Two suggested models were tested to explore the structural relationships between CIPP components. The first model represents the original CIPP model with indirect relationships between the four components: CIPP. The second model suggests direct relationships between the first three components (CIP) and the objectives or products (P).
24 February 2024 in European Journal of STEM Education
This study attempts to understand the source of variation in the Students’ Evaluation of Teaching (SET) of mathematics courses by comparing the data structure of engineering major students and mathematics major students’ datasets. The sample considered in this study consists of 644 students distributed into two majors: 237 mathematics students and 407 engineering students who filled out a 20-item SET questionnaire to rate the teaching practices of three different mathematics courses offered by the Department of Mathematics. The hypothesis tested in this study is: that variation in students’ perceptions of mathematics course teaching practices is different based on students’ majors (Mathematics versus Engineering). Measurement invariance (MI) analyses were used to examine the source of variation in the datasets and to compare engineering and mathematics students’ perceptions of the teaching effectiveness of mathematics courses. While the results of this study provide evidence of the SET validity, it was found that engineering students differently perceive three out of twenty of the SET questionnaire items when compared with mathematics major students.
2 January 2024 in Journal of Applied Research in Higher Education
Purpose: This study aims at assessing item fairness in students' evaluation of teaching based on students' academic college using measurement invariance analysis (MI).Design/methodology/approach: The sample of this study consists of 17,270 undergraduate students from 12 different academic colleges. SET survey consists of 20 Likert-type items distributed to four factors: planning, instruction, management and assessment was used to collect the data. The Lavaan R package with confirmatory factor analysis (CFA) was used to evaluate measurement invariance (MI). Four models of CFA were investigated and assessed: the configural model, the metric model, the scalar model and the residual invariance model. ANOVA was used to test the differences in SET according to academic colleges.Findings: MI analysis showed that the four levels of MI models are supported. ANOVA test showed that means of SET total scores are statistically different according to students' academic colleges. College of “Education” has the highest SET mean (88.64 out of 100), and all the differences between the College of Education’s SET mean and other colleges' SET means are statistically significant.Practical implications: The study recommends that higher education institutions test the MI of SET according to academic colleges and then use colleges with the highest SET at the university level as internal benchmarking to develop and enhance their teaching practices.Originality/value: This study is probably the only study that tested MI according to students' colleges before testing the differences between colleges in SET. If MI is not supported, then the comparisons between academic colleges are not applicable.
10 November 2023 in International Journal of Statistics in Medical Research
Introduction: Many research studies have shown that students' evaluations of teaching(SET) are affected by different variables without testing the requirement of fair comparisons. These studies have not tested the measurement equivalency of SET surveys according to these variables. Measurement equivalency of SET refers to whether a SET survey is interpreted similarly across different groups of individuals (Variable Levels). Without evidence of measurement invariance across different variables under investigation, the SET ratings should not be compared across these variables and this is the goal of this study. Methods: Measurement Invariance analysis of SET survey was investigated using 1649 responses to SET of four different medical core courses offered by the College of Science and College of Medicine and from different levels. Results: The results showed the existence of teaching practices in the SET survey that are not equivalently loaded on its factor across the levels of targeted variables, and the college offered medical courses were a source of variation in basic medical science students’ evaluation of teaching effectiveness. On the other hand, teaching practices in the SET survey are equivalently loaded on its factor across course levels. Discussion: The study results showed that the SET of medical courses is comparable to the courses only taught by the College of Medicine. These results provide evidence that medical courses are different from other courses offered by other colleges. This means that comparing SET of the College of Medicine with other colleges and colleges of medicine needs to compare SET results at the college level only.
14 October 2024 in Journal of Applied Research in Higher Education
Purpose: This study explores the distribution of insufficient effort responders according to methods of classifying students’ evaluation of teaching effectiveness in higher education. Five different methods were found in the literature to classify students’ evaluation of teaching effectiveness in higher education. Design/methodology/approach: Quantitative research methodology was used to achieve the goals of this study. Data from a major public university was used through 20 five-point items that are designed to measure students’ evaluation of teaching effectiveness. The dataset that consisted of 26,679 surveys was analyzed. Detecting insufficient efforts responding was based on item response theory procedures. Findings: The results show that insufficient effort responders are distributed differently to students’ evaluation of teaching effectiveness in higher education levels according to different methods of classifying these levels. The results of this study suggest using a percentage of students’ agreement of 4 or 5 for each item to classify SET levels and deleting IERs before interpreting SET results. Research limitations/implications: According to the results of this study, it is recommended to research the relationships between IER and SET scores and students’ motivation to participate in evaluating teaching effectiveness. Practical implications: According to the results of this study, it is recommended to:1– Exclude the IERs from the dataset before generating SET reports. 2– Use the percentage of 4 (agree) and 5 (strongly agree) satisfactions of SET items to classify and interpret SET results. Originality/value: Reviewing the literature shows the absence of studies that explore the distribution of insufficient effort responders according to methods of classifying students’ evaluation of teaching effectiveness in higher education. The results suggest using a percentage of students’ agreement of 4 or 5 for each item to classify SET levels and deleting IERs before interpreting SET results.