The Validity of the Malaysian Teachers’ Global Competency Level Instrument Using Cohen Kappa, Content Validity Ratio and Content Validity Index Analyses
https://doi.org/10.35877/454RI.asci3848
Keywords:
content validity, face validity, global competency, teachers
Viewed = 0 time(s)
Viewed = 0 time(s)
Abstract
Highly competent teachers are vital in developing globally competent individuals. There is a lack of empirical evidence supporting the validity aspect of the Malaysian Teachers' Global Competency Level Instrument. This study aims to evaluate face and content validity. A survey research design with a quantitative approach was conducted. It involves two experts for face validity and eight experts for content validity using purposive sampling techniques. For face validity, the two experts appointed were Malay language teachers with more than five years of teaching experience. To assess content validity, eight experts, including four professionals in measurement, evaluation, and global competency, and four field practitioners: teacher educators, and outstanding teachers. The instrument consists of 73 items with four constructs: self-awareness, global awareness, attitudes & values, and skills. The analysis involved Cohen’s Kappa for face validity, while content validity involved CVR and CVI. The results showed that face validity was (N=2, k=0.640), while for content validity, (N=8, CVI=0.95, and CVR=2 items refined). This instrument demonstrates strong validity as a measurement tool for the global competency level of Malaysian teachers. Further studies are recommended to be conducted: employing an advanced statistical analysis, like the Rasch Model, to enhance higher-quality items.
Downloads
Download data is not yet available.
References
Almanasreh, E., Moles, R., & Chen, T. F. (2019). Evaluation of methods used for estimating content validity. Research in Social and Administrative Pharmacy, 15(2), 214-221. https://doi.org/10.1016/j.sapharm.2018.03.066
Amatan, M. A., Han, C. G. K., & Pang, V. (2021). Kesahan kandungan soal selidik faktor konteks, input dan proses terhadap penerimaan pelaksanaan elemen pendidikan STEM dalam pengajaran dan pembelajaran guru menggunakan nisbah kesahan kandungan (CVR). International Journal of Advanced Research in Future Ready Learning and Education, 23(1), 10–22. https://akademiabaru.com/submit/index.php/frle/article/view/3923
Anggraini, F. D. P., Aprianti, A., Muthoharoh, N. A., IPermatasari, I., & Azalia, J. L. (2023). Validity and reliability questionnaire test of knowledge, attitudes, and behavior on dengue fever prevention. International Journal on Health and Medical Sciences, 1(2), 46–54.
Anizar, Gani, A., Khaldun, I., & Bahi, M. (2018). The development of a module with Microsoft Excel-based interactive media on the topic of buffer solution. Journal of Physics: Conference Series, 1088, 1-6. https://doi.org/10.1088/1742-6596/1088/1/012119
Association of College and Research Libraries. (2015). Framework for Information Literacy for Higher Education. American Library Association. https://www.ala.org/acrl/standards/ilframework
Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79–86. https://doi.org/10.1177/0748175613513808
Baco, S., & Ishak, M. Z. (2021). Kesahan dan kebolehpercayaan instrumen penilaian kendiri pembelajaran ungkapan algebra tingkatan dua. Malaysian Journal of Social Sciences and Humanities (MJSSH), 6(11), 127–137. https://doi.org/10.47405/mjssh.v6i11.1136
Bhattacharyya, S., Kaur, R., Kaur, S., & Ali, S. A. (2017). Validity and reliability of a questionnaire: A literature review. Chronicles of Dental Research, 6(2), 17–24.
Bolarinwa, O. A. (2015). Principles and methods of validity and reliability testing of questionnaires used in social and health science research. The Nigerian Postgraduate Medical Journal, 22(4), 195–201. https://doi.org/10.4103/1117-1936.173959
Cohen, L., Manion, L. & Morrison, K. 2018. Research Methods in Education. 6th ed. New York: Routledge.
Darusalam, G., & Hussin, S. (2021). Metodologi penyelidikan dalam pendidikan: Amalan dan analisis kajian. Penerbit Universiti Malaya.
Dettori, J. R., & Norvell, D. C. (2020). Kappa and beyond: Is there agreement? Global Spine Journal, 10(4), 499–501.
Gay, L. R., Mills, G. E., & Airasian, P. (2012). Educational research: Competencies for analysis and applications (10th ed.). Merrill Prentice Hall.
Goh, P. S. C., & Ooi, P. K. (2020). Developing global competence in Malaysian universities: Challenges and possible solutions. International Journal of Higher Education, 8(4), 256–265.
GraphPad Software. (2025). Quantify agreement with kappa. Retrieved from https://www.graphpad.com/quickcalcs/kappa1/
Hair, J. F., Celsi, M. W., Ortinau, D. J., & Bush, R. P. (2013). Essentials of marketing research (3rd ed.). McGraw Hill.
Herlinawati H., Marwa M., Ismail N., Junaidi, Liza L.O., Situmorang D.D.B (2024). The integration of 21st century skills in the curriculum of education. Heliyon. 10(15): e35148. doi: 10.1016/j.heliyon. 2024.e35148.
Jiaxin, G., Huijuan, Z., & Hasan, M. H. (2024). Global competence in higher education: A ten-year systematic literature review. Frontiers in Education, 9, Article 1404782. https://doi.org/10.3389/feduc.2024.1404782
Kamaluddin, M. R., Nasir, R., Wan Sulaiman, W. S., Khairudin, R., & Ahmad Zamani, Z. (2017). Validity and psychometric properties of Malay translated religious orientation scale-revised among Malaysian adult samples. Akademika, 87(2), 133–144.
Kamaruddin, M., & Mohd Matore, M. E. E. (2021). Development and Validation of Psychometric Properties of the 10 IB Learner Profile Instrument (10IBLP-I): A Combination of the Rasch and Classical Measurement Model. International Journal of Environmental Research and Public Health, 18(12), 6455. https://doi.org/10.3390/ijerph18126455
Kopish, M. (2016). Preparing globally competent teacher candidates through cross-cultural experiential learning. Journal of Social Studies Education Research, 7(2), 75–108.
Landis, J.R. & Koch, G.G. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1): 159-174.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575.
Lindell, M. K., & Brandt, C. J. (1999). Assessing interrater agreement on the job relevance of a test: A comparison of CVI, T, rWG(J)g and rWG(J)g indexes. Journal of Applied Psychology, 84(4), 640–647. https://doi.org/10.1037/0021-9010.84.4.640
Luqiu, L. R. (2019). In the age of misinformation: The importance of information literacy. In T. T. C. Tan & M. Y. Fee (Eds.), Information experience in theory and design (pp. 199–205). Springer. https://doi.org/10.1007/978-981-13-6681-9_15
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–386.
Mohd Matore, M.E.E, Zainal, M.A., Mohd Noh, M.F., Khairani, A.Z., & Abd Razak, N. (2021)
The Development and Psychometric Assessment of Malaysian Youth Adversity Quotient Instrument (MY-AQi) by Combining Rasch Model and Confirmatory Factor Analysis. IEEE Access, 9, 13314–13329
Masdoki, M., Din, R. & Mohd Matore M.E.E (2021) Teaching 4.0 competency in higher learning institutions: A systematic mapping review. International Journal of Learning, Teaching and Educational Research. 20 (10), 217–231
Mohamat, R., Sumintono, B., & Abd Hamid, H. R. (2022). Analisis kesahan kandungan instrumen kompetensi guru untuk melaksanakan pentaksiran bilik darjah menggunakan Model Rasch pelbagai faset. Jurnal Pendidikan Malaysia, 47(1), 1–14. https://doi.org/10.17576/JPEN-2022-47.01-01
Mohd Noh, M.F & Mohd Matore M.E.E (2022) Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis. Frontiers in Psychology, 13:941084. doi: 10.3389/fpsyg.2022.941084
Mopara, R., & Sanrattana, W. (2023). Developing teachers to develop students' 21st century skills. World Journal of Education, 13(3), 94–104. https://doi.org/10.5430/wje.v13n3p94
Oktari, R. S., Comfort, L. K., Syamsidik, & Dwitama, P. (2020). Measuring coastal cities’ resilience toward coastal hazards: Instrument development and validation. Progress in Disaster Science, 5, Article 100065.
Oluwatayo, J. A. (2012). Validity and reliability issues in educational research. Journal of Educational and Social Research, 2(2), 391–400.
Omar, A., Hamzah, S. A., & Kee, C. P. (2021). Kesahan dan kebolehpercayaan instrumen batas etnik di sekolah kebangsaan Zon Utara Semenanjung Malaysia. e-BANGI: Jurnal Sains Sosial dan Kemanusiaan, 18(3), 253–269.
Organization for Economic Cooperation and Development (OECD). (2018). PISA 2018 global competence. Retrieved from https://www.oecd.org/en/topics/sub-issues/global-competence/pisa-2018-global-competence.html
Parmigiani, D., Jones, S., Silvaggio, C., Nicchia, E., Ambrosini, A., Pario, M., et al. (2022). Assessing global competence within teacher education programs: How to design and create a set of rubrics with a modified Delphi method. SAGE Open, 12(4), Article 21582440221128794. https://doi.org/10.1177/21582440221128794
Parmigiani, D., Bar Nir, A., Ferguson-Patrick, K., Forkosh-Baruch, A., Heddy, E., Impedovo, M. A., et al. (2023). Assessing the development of global competence in teacher education programmes: Internal consistency and reliability of a set of rubrics. Higher Education Pedagogies, 8(1), Article 2216190. https://doi.org/10.1080/23752696.2023.2216190
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health, 30(4), 459–467.
Potter, W. (2022). Analysis of definitions of media literacy. Journal of Media Literacy Education, 14(2), 27-43. https://doi.org/10.23860/JMLE-2022-14-2-3
Ramaila, S., & Molwele, A. J. (2022). The role of technology integration in the development of 21st century skills and competencies in life sciences teaching and learning. International Journal of Higher Education, 11(5), 9–17. https://doi.org/10.5430/ijhe.v11n5p9
Rau, G., & Shih, Y. S. (2021). Evaluation of Cohen’s kappa and other measures of inter-rater agreement for genre analysis and other nominal data. Journal of English for Academic Purposes, 53, Article 101021. https://doi.org/10.1016/j.jeap.2021.101021
Redecker, C. (2017). European framework for the digital competence of educators: DigCompEdu. Publications Office of the European Union. https://doi.org/10.2760/159770
Sabharwal, C. L. (2021). Cohen’s kappa statistic and new kappa statistic for measuring and interpreting inter-rater agreement. International Journal of Research in Engineering and Science, 9(7), 23–28.
Sabtu, S. H., Matore, M. E. M., & Maat, S. M. (2025). Enhancing face validity in teaching quality with Six Sigma approach for sustainable development goals (SDG). SDGsReview, 5, e03335. https://doi.org/10.47172/2965-730X.SDGsReview.v5.n02.pe03335
Taherdoost, Hamed. (2016). Validity and Reliability of the Research Instrument; How to Test the Validation of a Questionnaire/Survey in a Research. International Journal of Academic Research in Management, 5(3), 28-36. https://doi.org/10.2139/ssrn.3205040.
Wang, B., Rau, P. L. P., & Yuan, T. (2023). Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology, 42(9), 1324–1337. https://doi.org/10.1080/0144929X.2022.2072768
Yaghmaie, F. (2003). Content validity and its estimation. Journal of Medical Education, 3(1),25-27.
Yeşilyurt, E., Vezne, R. (2023). Digital literacy, technological literacy, and internet literacy as predictors of attitude toward applying computer-supported education. Education and Information Technologies. 28, 9885–9911. https://doi.org/10.1007/s10639-022-11311-1
Zainal, M. A., Mohd Matore, M. E. E., W Musa, W. N., & Hashim, N. H. (2020). Kesahan kandungan instrumen pengukuran tingkah laku inovatif guru menggunakan kaedah nisbah kesahan kandungan (CVR). Akademika, 90, 43-54. https://ejournals.ukm.my/akademika/article/view/42052
Zhang, Z., & Garcia, L. (2023). Examining dimensionality and validity of the academic integrity survey instrument. Journal of Education and Development, 7(1), 46–54. https://doi.org/10.20849/jed.v7i1.1326
Amatan, M. A., Han, C. G. K., & Pang, V. (2021). Kesahan kandungan soal selidik faktor konteks, input dan proses terhadap penerimaan pelaksanaan elemen pendidikan STEM dalam pengajaran dan pembelajaran guru menggunakan nisbah kesahan kandungan (CVR). International Journal of Advanced Research in Future Ready Learning and Education, 23(1), 10–22. https://akademiabaru.com/submit/index.php/frle/article/view/3923
Anggraini, F. D. P., Aprianti, A., Muthoharoh, N. A., IPermatasari, I., & Azalia, J. L. (2023). Validity and reliability questionnaire test of knowledge, attitudes, and behavior on dengue fever prevention. International Journal on Health and Medical Sciences, 1(2), 46–54.
Anizar, Gani, A., Khaldun, I., & Bahi, M. (2018). The development of a module with Microsoft Excel-based interactive media on the topic of buffer solution. Journal of Physics: Conference Series, 1088, 1-6. https://doi.org/10.1088/1742-6596/1088/1/012119
Association of College and Research Libraries. (2015). Framework for Information Literacy for Higher Education. American Library Association. https://www.ala.org/acrl/standards/ilframework
Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79–86. https://doi.org/10.1177/0748175613513808
Baco, S., & Ishak, M. Z. (2021). Kesahan dan kebolehpercayaan instrumen penilaian kendiri pembelajaran ungkapan algebra tingkatan dua. Malaysian Journal of Social Sciences and Humanities (MJSSH), 6(11), 127–137. https://doi.org/10.47405/mjssh.v6i11.1136
Bhattacharyya, S., Kaur, R., Kaur, S., & Ali, S. A. (2017). Validity and reliability of a questionnaire: A literature review. Chronicles of Dental Research, 6(2), 17–24.
Bolarinwa, O. A. (2015). Principles and methods of validity and reliability testing of questionnaires used in social and health science research. The Nigerian Postgraduate Medical Journal, 22(4), 195–201. https://doi.org/10.4103/1117-1936.173959
Cohen, L., Manion, L. & Morrison, K. 2018. Research Methods in Education. 6th ed. New York: Routledge.
Darusalam, G., & Hussin, S. (2021). Metodologi penyelidikan dalam pendidikan: Amalan dan analisis kajian. Penerbit Universiti Malaya.
Dettori, J. R., & Norvell, D. C. (2020). Kappa and beyond: Is there agreement? Global Spine Journal, 10(4), 499–501.
Gay, L. R., Mills, G. E., & Airasian, P. (2012). Educational research: Competencies for analysis and applications (10th ed.). Merrill Prentice Hall.
Goh, P. S. C., & Ooi, P. K. (2020). Developing global competence in Malaysian universities: Challenges and possible solutions. International Journal of Higher Education, 8(4), 256–265.
GraphPad Software. (2025). Quantify agreement with kappa. Retrieved from https://www.graphpad.com/quickcalcs/kappa1/
Hair, J. F., Celsi, M. W., Ortinau, D. J., & Bush, R. P. (2013). Essentials of marketing research (3rd ed.). McGraw Hill.
Herlinawati H., Marwa M., Ismail N., Junaidi, Liza L.O., Situmorang D.D.B (2024). The integration of 21st century skills in the curriculum of education. Heliyon. 10(15): e35148. doi: 10.1016/j.heliyon. 2024.e35148.
Jiaxin, G., Huijuan, Z., & Hasan, M. H. (2024). Global competence in higher education: A ten-year systematic literature review. Frontiers in Education, 9, Article 1404782. https://doi.org/10.3389/feduc.2024.1404782
Kamaluddin, M. R., Nasir, R., Wan Sulaiman, W. S., Khairudin, R., & Ahmad Zamani, Z. (2017). Validity and psychometric properties of Malay translated religious orientation scale-revised among Malaysian adult samples. Akademika, 87(2), 133–144.
Kamaruddin, M., & Mohd Matore, M. E. E. (2021). Development and Validation of Psychometric Properties of the 10 IB Learner Profile Instrument (10IBLP-I): A Combination of the Rasch and Classical Measurement Model. International Journal of Environmental Research and Public Health, 18(12), 6455. https://doi.org/10.3390/ijerph18126455
Kopish, M. (2016). Preparing globally competent teacher candidates through cross-cultural experiential learning. Journal of Social Studies Education Research, 7(2), 75–108.
Landis, J.R. & Koch, G.G. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1): 159-174.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575.
Lindell, M. K., & Brandt, C. J. (1999). Assessing interrater agreement on the job relevance of a test: A comparison of CVI, T, rWG(J)g and rWG(J)g indexes. Journal of Applied Psychology, 84(4), 640–647. https://doi.org/10.1037/0021-9010.84.4.640
Luqiu, L. R. (2019). In the age of misinformation: The importance of information literacy. In T. T. C. Tan & M. Y. Fee (Eds.), Information experience in theory and design (pp. 199–205). Springer. https://doi.org/10.1007/978-981-13-6681-9_15
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–386.
Mohd Matore, M.E.E, Zainal, M.A., Mohd Noh, M.F., Khairani, A.Z., & Abd Razak, N. (2021)
The Development and Psychometric Assessment of Malaysian Youth Adversity Quotient Instrument (MY-AQi) by Combining Rasch Model and Confirmatory Factor Analysis. IEEE Access, 9, 13314–13329
Masdoki, M., Din, R. & Mohd Matore M.E.E (2021) Teaching 4.0 competency in higher learning institutions: A systematic mapping review. International Journal of Learning, Teaching and Educational Research. 20 (10), 217–231
Mohamat, R., Sumintono, B., & Abd Hamid, H. R. (2022). Analisis kesahan kandungan instrumen kompetensi guru untuk melaksanakan pentaksiran bilik darjah menggunakan Model Rasch pelbagai faset. Jurnal Pendidikan Malaysia, 47(1), 1–14. https://doi.org/10.17576/JPEN-2022-47.01-01
Mohd Noh, M.F & Mohd Matore M.E.E (2022) Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis. Frontiers in Psychology, 13:941084. doi: 10.3389/fpsyg.2022.941084
Mopara, R., & Sanrattana, W. (2023). Developing teachers to develop students' 21st century skills. World Journal of Education, 13(3), 94–104. https://doi.org/10.5430/wje.v13n3p94
Oktari, R. S., Comfort, L. K., Syamsidik, & Dwitama, P. (2020). Measuring coastal cities’ resilience toward coastal hazards: Instrument development and validation. Progress in Disaster Science, 5, Article 100065.
Oluwatayo, J. A. (2012). Validity and reliability issues in educational research. Journal of Educational and Social Research, 2(2), 391–400.
Omar, A., Hamzah, S. A., & Kee, C. P. (2021). Kesahan dan kebolehpercayaan instrumen batas etnik di sekolah kebangsaan Zon Utara Semenanjung Malaysia. e-BANGI: Jurnal Sains Sosial dan Kemanusiaan, 18(3), 253–269.
Organization for Economic Cooperation and Development (OECD). (2018). PISA 2018 global competence. Retrieved from https://www.oecd.org/en/topics/sub-issues/global-competence/pisa-2018-global-competence.html
Parmigiani, D., Jones, S., Silvaggio, C., Nicchia, E., Ambrosini, A., Pario, M., et al. (2022). Assessing global competence within teacher education programs: How to design and create a set of rubrics with a modified Delphi method. SAGE Open, 12(4), Article 21582440221128794. https://doi.org/10.1177/21582440221128794
Parmigiani, D., Bar Nir, A., Ferguson-Patrick, K., Forkosh-Baruch, A., Heddy, E., Impedovo, M. A., et al. (2023). Assessing the development of global competence in teacher education programmes: Internal consistency and reliability of a set of rubrics. Higher Education Pedagogies, 8(1), Article 2216190. https://doi.org/10.1080/23752696.2023.2216190
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health, 30(4), 459–467.
Potter, W. (2022). Analysis of definitions of media literacy. Journal of Media Literacy Education, 14(2), 27-43. https://doi.org/10.23860/JMLE-2022-14-2-3
Ramaila, S., & Molwele, A. J. (2022). The role of technology integration in the development of 21st century skills and competencies in life sciences teaching and learning. International Journal of Higher Education, 11(5), 9–17. https://doi.org/10.5430/ijhe.v11n5p9
Rau, G., & Shih, Y. S. (2021). Evaluation of Cohen’s kappa and other measures of inter-rater agreement for genre analysis and other nominal data. Journal of English for Academic Purposes, 53, Article 101021. https://doi.org/10.1016/j.jeap.2021.101021
Redecker, C. (2017). European framework for the digital competence of educators: DigCompEdu. Publications Office of the European Union. https://doi.org/10.2760/159770
Sabharwal, C. L. (2021). Cohen’s kappa statistic and new kappa statistic for measuring and interpreting inter-rater agreement. International Journal of Research in Engineering and Science, 9(7), 23–28.
Sabtu, S. H., Matore, M. E. M., & Maat, S. M. (2025). Enhancing face validity in teaching quality with Six Sigma approach for sustainable development goals (SDG). SDGsReview, 5, e03335. https://doi.org/10.47172/2965-730X.SDGsReview.v5.n02.pe03335
Taherdoost, Hamed. (2016). Validity and Reliability of the Research Instrument; How to Test the Validation of a Questionnaire/Survey in a Research. International Journal of Academic Research in Management, 5(3), 28-36. https://doi.org/10.2139/ssrn.3205040.
Wang, B., Rau, P. L. P., & Yuan, T. (2023). Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology, 42(9), 1324–1337. https://doi.org/10.1080/0144929X.2022.2072768
Yaghmaie, F. (2003). Content validity and its estimation. Journal of Medical Education, 3(1),25-27.
Yeşilyurt, E., Vezne, R. (2023). Digital literacy, technological literacy, and internet literacy as predictors of attitude toward applying computer-supported education. Education and Information Technologies. 28, 9885–9911. https://doi.org/10.1007/s10639-022-11311-1
Zainal, M. A., Mohd Matore, M. E. E., W Musa, W. N., & Hashim, N. H. (2020). Kesahan kandungan instrumen pengukuran tingkah laku inovatif guru menggunakan kaedah nisbah kesahan kandungan (CVR). Akademika, 90, 43-54. https://ejournals.ukm.my/akademika/article/view/42052
Zhang, Z., & Garcia, L. (2023). Examining dimensionality and validity of the academic integrity survey instrument. Journal of Education and Development, 7(1), 46–54. https://doi.org/10.20849/jed.v7i1.1326
Published
2025-04-30
Issue
Section
Articles
How to Cite
Ibrahim, S. N. A., & Mohd Matore, M. E. E. (2025). The Validity of the Malaysian Teachers’ Global Competency Level Instrument Using Cohen Kappa, Content Validity Ratio and Content Validity Index Analyses. Journal of Applied Science, Engineering, Technology, and Education, 7(1), 82-93. https://doi.org/10.35877/454RI.asci3848
Copyright (c) 2025 Siti Nurul Aqiedah Ibrahim, Mohd Effendi Ewan Mohd Matore (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
