Professor George Engelhard, Jr., Ph.D. joined the faculty at The University of Georgia in the fall of 2013. He is professor emeritus at Emory University (1985 to 2013). Professor Engelhard received his Ph.D. in 1985 from The University of Chicago (MESA Program--measurement, evaluation, and statistical analysis). While at the University of Chicago, he worked with Professors Ben Bloom and Ben Wright. In 2015, he received the first Qiyas Award for Excellence in International Educational Assessment recognizing his contributions to the field of education based on his book (Invariant measurement: Using Rasch models in the social, behavioral, and health sciences), as well as his program of research that focuses on the improvement of educational measurement at the local, national, and international levels. He was the 2018 recipient of Benjamin Drake Wright Senior Scholar Award from American Educational Research Association. He is an elected fellow of the American Educational Research Association. He is currently president of the Pacific Rim Objective Measurement Society (2022-2023). He recently published with Jue Wang (The University of Miami) a book entitled: Rasch models for solving measurement problems: Invariant measurement in the social sciences.

Dr. Che Kuong Hon, is the President of the World Sports University. Born on August 5, 1954, in Macau, he holds a Doctor of Laws (LL.D.) degree from Xiamen University and a Master of Laws degree from Asia International Open University. He has also completed various diploma and bachelor's degree programs in law, business administration, and Chinese laws. Dr. Che has an extensive teaching career that spans over 30 years, including serving as a professor at Zhejiang Police Academy, Macau Public Security Academy, and Qilu University, among others. He has also held various positions in social organisations and international sports federations, including Chairman of EC for the General Association of World Sports Federations and Vice President of the International Sport Network Organisation. Hon is an 8th Dan holder in Judo, an international instructor, and an international referee.

Professor Luo held a doctorate degree of philosophy in Mathematical Psychology obtained from Murdoch University in 1995. Professor Luo was the Director – Assessment Technology and Research (D-ATR) at the HKEAA and had been serving the authority for almost 15 years, leading the team at the HKEAA responsible for the design and development of the procedures and programs of the HKDSE Examinations. Before his stint at the HKEAA, Dr. Luo had held various academic positions as professor and adjunct professor at a number of universities in the Mainland of the People’s Republic of China, Singapore and Australia. He is now a professor in South China Normal University and Jiangxi Normal University.  His major area of research includes development of various psychometric models and parameter estimation algorithms for achievement and attitude measurements. His publications are internationally recognised, in particular the computer programs for test data analysis and processing developed/co-developed by him are widely used by numerous major examination organisations and academic researchers in the education industry globally.

Dr Xiaoting Huang is a tenured associate professor in the Graduate School of Education at Peking University. Prior to joining PKU, she was Director- Examinations, Assessment and Research of the Hong Kong Examinations and Assessment Authority. Her research area spans a range of issues from test reliability and validity, item response modeling, computational psychometrics, to policy issues on the use of assessment data. Dr. Huang’s speech is about taking the Computational Psychometrics Approach to investigate University Student’s Collaborative Problem Solving Competency. According to Dr. Huang, collaborative problem solving is a key competency for innovative talents. However, assessment data on this high-order complex competency is not available due to technical difficulties. As a result, the varying effects of different reform strategies are yet to be disentangled. This study aims to design a new instrument taking the Computational Psychometrics approach. Specifically, item response models and graph neural network methods will be used jointly to develop an automated scoring system so that the multimodal interaction data during collaborative problem solving can be analyzed scientifically and efficiently.

Jue Wang, Ph.D. is currently a professor in Department of Psychology at The University of Science and Technology of China. Dr. Wang received her Ph.D. in Quantitative Methodology Program under Educational Psychology at The University of Georgia, and previously worked in Research, Measurement & Evaluation Program at The University of Miami. Her research focuses on examining rater effects in rater-mediated assessments using Rasch measurement models and unfolding models. She has published in peer-reviewed journals including Educational Psychology Review, Psychology of Aesthetics, Creativity, and the Arts, Educational and Psychological Measurement, Journal of Educational Measurement, and Assessing Writing. Dr. Wang has also co-authored a book (with Professor George Engelhard) entitled Rasch models for solving measurement problems: Invariant measurement in the social sciences published by Sage as part of Quantitative Applications in the Social Sciences (QASS) series.

Yaru Meng is a professor at School of Foreign Studies, Xi’an Jiaotong University, China. She is a Council Member of China Association for Language Testing and Assessment. Her research interests cover Language Assessment, Cognitive Diagnostic Assessment, Dynamic Assessment and Language Teaching & Technology. She was once a visiting scholar at the Pennsylvania State University and University of Maryland. She won 10 national and provincial research funding from China National Social Science Fund, China Ministry of Education, Shaanxi Province Social Fund. She published widely on international journals like Modern Language Journal, Journal of Educational Measurement, System, Frontiers in Psychology, and local ones like Modern Foreign Languages, Foreign Language Education and Foreign Language World. She presented her studies at around 35 conferences home and abroad.

About the talk: Cognitive Diagnostic Models (CDMs) are multidimensional latent variable models for cognitive diagnosis. They have also demonstrated their advantages in English as Foreign Language (EFL) listening in identifying learners’ cognitive processes in test items, and offering feedback of their strengths and weaknesses. This is achieved by adopting appropriate CDMs to analyze Q-matrix (of test item and the examined attributes) and learners’ responses of 0/1 (“0” denoting wrong answer and “1” correct answer). However, most studies undervalue the rich information of students’ incorrect options (in the case of multiple choice questions, MCQ) which are intentionally designed to diagnose learners’ misconceptions, barriers or bugs (in psychometric term), and help to determine the prevalence and distribution of these bugs across populations. This diagnosis help teachers to make well-informed decisions about remediation supports at both individual and group level. 

In the current study, five EFL listening barrier attributes were first identified and two Bug Q-matrices were developed so as to comparatively analyze the learner’s responses with different Bug-CDMs. The results revealed that Bug-GDINA was the optimal model, and the most prevalent barriers were identified and validated. Besides, the interactions of barriers demonstrated both compensatory and non-compensatory relationships in causing listening comprehension failures. The study proved the feasibility of Bug-CDMs (Bug-GDINA in particular) in diagnosing listening problems from the incorrect options. Limitations and potentials for future research are also proposed.

Keywords: Cognitive diagnostic Models (CDMs), EFL listening barriers, Incorrect options, Bug-GDINA