Introduction: This study analyzed the threshold-exceedance performance of human and AI scoring agreement measures for scoring of open-ended items. Monte Carlo simulation was performed to represent the different types of errors encountered in automatic scoring determined by using studies in the literature. Accordingly, an examination was conducted of the agreement between human and artificial intelligence ratings under differing error conditions. The objective of the research was to ascertain the statistical power levels achieved by disparate agreement metrics under varying conditions. The objective of this research was to ascertain which metric would prove to be the most effective method, and under which conditions. Methods: Data with conditions including systematic additive bias, variance inflation, midpoint compression, class imbalance, and subgroup-related offsets. Human scores served as a reference to assess agreement. Agreement levels were evaluated with ICC(A,1), Krippendorff’s α (ranked), quadratic weighted kappa (QWK), and Bland-Altman along with tolerance-based agreement metrics. Threshold-exceedance performance was defined as the proportion with which each metric surpassed conventional adequacy standards. Analyses were also conducted on real data to validate the analyses conducted in the second part of the study. In this part, written texts were scored by six different students using three teachers and two large language models. Results: ICC(A,1) shown higher threshold-exceedance performance for low and moderate variance inflation. QWK was observed to reach a moderate level of robustness. Krippendorff’s a showed consistent performance, especially in conditions where the distributions were unbalanced or variance inflated. Tolerance-based fit demonstrated numerical closeness between human and AI scores. The findings showed patterns consistent with simulated impairments. Discussion: All findings indicate that fit indices vary systematically across different structural mechanisms and sampling conditions. The results suggest that different conditions can affect the interpretability of automated scores. Accordingly, the need for multi-metric assessment frameworks when assessing human-AI score fit is highlighted.

Bu çalışmada istatistiksel varsayımlar açısından farklı olan yöntemler kullanılarak aynı veri setinin ölçme değişmezliği ile ilgili sonuçların incelenmesi amaçlanmıştır. Ayrıca normallik varsayımını gerektiren ve gerektirmeyen yöntemlerin normallik varsayımı sağlanamayan durumlarda farklı sonuçlar gösterip göstermediğine bakılmıştır. Bu amaca göre PISA 2012 alt ölçeklerinden beş maddeden oluşan Problem Çözmeye Açıklık ölçeği Türkiye ve Finlandiya örneklemleri veri seti üzerinde yapısal eşitlik modellemesi çatısı altındaki ortalama kovaryans yapılarının değişmezliği analizi ve örtük sınıf analizi çatısı altındaki çoklu grup örtük sınıf analizi yöntemi analizler gerçekleştirilmiştir. Ortalama kovaryans yapılarının değişmezliği analizi için Lisrel 8.72; çoklu grup örtük sınıf analizi için ise Latent Gold 5.1 programları kullanılmıştır. Ortalama kovaryans yapılarının değişmezliği analizi için yapısal değişmezlik ile başlayan ve katı değişmezlik aşaması ile biten aşamalı test etme yöntemi adımları takip edilmiştir. Örtük sınıf analizi için ise örtük sınıf sayısının belirlenmesinden sonra heterojen, kısmi homojen ve homojen model test edilmiştir. Analizlerden önce Kolmogorv Smirnov testi kullanılarak her bir maddenin normalliği incelenmiştir. Alt örneklemler ve tüm grup için hiç bir madde normal dağılım göstermemiştir. Normallik varsayımı gerektiren ortalama kovaryans yapılarının değişmezliği analizi sonuçlarına göre katı değişmezlik kabul edilmiştir. Bir başka deyişle ölçme değişmezliği sağlanmıştır. Fakat normallik varsayımı gerektirmeyen çok gruplu örtük sınıf analizi için kısmi homojen model kabul edilmiştir. Kısmi homojen model ortalama kovaryans yapılarının analizi adımlarından zayıf değişmezliğe karşılık gelmektedir. Elde edilen sonuçlara göre varsayımlar açısından farklılaşan metotlar kullanıldığında ölçme değişmezliği bulguları değişiklik göstermiştir. Bu bağlamda yöntemler için gerekli varsayımlar mutlaka incelenmeli ve gizil ve gözlenen değişken yapıları göz önünde bulundurularak uygun yöntem seçmeye dikkat edilmelidir.

It is necessary to examine the measurement invariance (MI) among groups in studies where different groups are compared by using a measurement instrument. Most of the studies, measurement invariance is tested with multiple group confirmatory factor analysis. This model applies many model adjustments based on the modification indexes. Therefore, it is not practical due to too many large modification indexes while testing MI over many groups. Besides scalar model is a poor model fit when comparing many groups and so does not hold MI. In this study, the aim is to explain the basic concepts and processes of the alignment method which is offered as a new method for testing MI and illustrate an application on the real data set. In this study, measurement invariance among 56 countries including Turkey is tested with alignment method in order to set an example for researchers. For this purpose, the Instrumental Motivation Scale data, which is one of the psychological measurement instruments used in PISA 2015, was used. As a result of MG-CFA, it was found that configural invariance was ensured. The fit indexes of CFI and TLI were calculated as 0.982 and 0.946 respectively in this stage. After that, metric invariance was tested by considering the difference of fit indices obtained for the two stages. It was found that the metric invariance could not be provided. Alignment results show which countries hold MI and which do not. Besides it provides information which items have the most invariants for groups that hold MI.

The aim of this study is to adapt the Application Based Smartphone Addiction Scale, which has been developed to determine the smart phone addiction, which is becoming a common problem every day. The study was carried out with 474 students in 2017 - 2018 academic year at Bolu Abant Izzet Baysal University Faculty of Education. In exploratory factor analysis (EFA) for construct validity, the items were collected under a single factor in keeping with the original structure. The rate of explained variance was 52.658%. The eigenvalue of the factor was determined to be 3.159. Factor loadings of the items ranged between 0.531 and 0.835, and all of the error variances were less than 0.05. Asymptotic covariance and correlation matrices and Weighted Least Square (WLS) estimation method were preferred because of the structure of data in Confirmatory Factor Analysis (CFA). T-values of the items were found to be significant at the level of 0.01 (30.522-41.257). Factor loadings of the items were found to be high (0.50-0.81). When the model fit indices were examined, the fit values calculated as χ2/sd = 2.09, RMSEA = 0.068, GFI = 0.99, AGFI = 0.98, CFI = 0.98, NNFI = 0.96, NFI = 0.96, SRMR = 0.044 indicated acceptable or excellent fit. Cronbach Alpha internal consistency coefficient was found to be 0.81 within the scope of reliability studies. In addition, the test-retest correlation coefficient was found to be 0.92 at four-week intervals. In this study, it has been ensured that this scale related to smartphone addiction, which has recently become a serious problem, has been introduced to national literature.

Validity is the most important psychometric feature that should be found in a measurement tool. Measurement equivalence is one of the evidences for the validity of measurement tools. Providing information to the researchers about the methods of identifying measurement equivalence may contribute to present a complete validity evidence. The purpose of this study is to compare the statistical power ratios of methods used for examining the measurement equivalence based on structural equation modeling and item response theory on the artificial data sets generated by diversifying variables of sample size, number of items, and the ratios of the items having differential item function. In accordance with this purpose, the variables have been varied to include three different levels. In the analysis, multi-group confirmatory factor analysis was used which is among the methods based on the structural equation modeling, and likelihood ratio test and comparison of the item parameters methods were used which are among the methods based on item response theory. Multi-group group confirmatory factor analysis (93,50%) and likelihood ratio test (96,75%) methods have reached the highest statistical power ratio in the condition that the sample size is 1000/1000, the number of items is 40 items and the ratios of the items having differential item function is 10%. The method of comparing the item parameters (94,50%) has reached the highest statistical power ratio in the condition of sample size is 1000/1000, the number of items is 20 and the ratios of the items having differential item function are 10-20%.