Measuring a national reading program: questionnaires design, validation and pilot testing

Abstract

During times of crisis, such as the COVID-19 pandemic, the evaluation of educational programs becomes crucial for making evidence-based decisions. This study aims to validate and pilot an assessment instrument tailored to evaluate an innovative national reading program in Chile, particularly during the critical phase of post-pandemic educational recovery. The focus of this study is to encompass three key areas: a. assessing reading proficiency in first-grade students, b. examining the utilization of technology and reading materials in early childhood education, and c. gauging the comprehension and utilization of program information by all stakeholders involved. The research design is grounded in a research and development (R&D) approach employing a mixed-method model. We engaged 11 experts in the field to assess the questionnaire's validity using the Delphi method, administered to a representative sample. Subsequently, through confirmatory factor analysis, we established the construct validity of the questionnaires, revealing goodness-of-fit indices such as comparative fit index (CFI) scores ranging from 0.90 to 0.92, a root means square error of approximation (RMSEA) of 0.08, and Tucker-Lewis’s index (TLI) values between 0.88 and 0.90. Reliability analysis yielded robust outcomes, with Cronbach's alpha coefficients ranging between 0.89 and 0.93 for the questionnaire and McDonald’s omega coefficients ranging between 0.92 and 0.93. Our study confirms the validity of the questionnaires, positioning them as dependable tools for evaluating the national reading literacy development program in Chile. These validated instruments hold the potential to facilitate further research endeavors and inform the formulation of effective public reading policies in Chile and beyond, across diverse educational landscapes.