Introduction

Background/rationale

Contemporary nursing education aims to prioritize student-centred learning, which is perceived as high-order, flexible, and individualized [1]. Recent advances in technology have accelerated educational innovations such as blended learning by providing easy access to information [2,3]. Blended learning is a novel student-centred pedagogical approach that includes technology-mediated online and face-to-face (F2F) learning [2]. The flipped classroom (FC) is one of several modern blended learning strategies [3]. In FC, teachers use technology to share pre-class learning material to activate low-order learning, and students study the material before attending the F2F classroom. Teachers design the F2F classroom as an interactive educational environment by using student-centred teaching strategies that allow the students to apply or evaluate the learnt concepts [1,2].

The FC has entered into use in undergraduate nursing education and has been in the spotlight in discussions of the implementation of nursing curricula [2,3]. With an increasing emphasis on the FC, the available empirical evidence on the usage and efficacy of the FC in nursing education mainly refers to its effects on students’ academic achievements and does not take into account other aspects of educational effectiveness [1,2]. Thorndike [4] in 1932 outlined a law of learning according to which students’ readiness to learn is an indispensable factor for measuring the degree of success of academic achievement. Readiness for the FC can be conceived of as a concept describing the ability of an individual to benefit from blended learning [5]. Moreover, readiness refers to a level of mental and physical preparedness among learners when taking part in the FC [6]. Assessing students’ readiness is a preliminary step for implementing the FC [3]. However, there is limited research investigating students’ readiness for the FC educational process in the context of nursing education. Therefore, it would be valuable for educational and research purposes to develop a measurement instrument to investigate students’ readiness for the FC.

Objectives

The aim of this study was to develop and validate a tool to measure nursing students’ readiness for the FC, namely the Nursing Students’ Readiness for Flipped Classroom (NSR-FC). Content validity, construct validity, and reliability tests were done to validate the measurement scale.

Methods

Ethics statement

The study was approved by the University of Auckland Human Participants Ethics Committee (reference no., 024079). Participants were provided with an information sheet before the anonymized questionnaire was administered. Participants were clearly informed that the voluntary return of the questionnaire to the collection box indicated their consent to participate in the anonymized survey.

Study design

This was a psychometric study to validate the measurement scale based on experts’ opinion and survey results for the scale.

Participants

Five experts participated in validity testing of the scale, and 265 undergraduate nursing students (in the 1st and 2nd academic years) from 3 state universities (Colombo University [n=141], University of Peradeniya [n=73], and Eastern University, Sri Lanka [n=51]) in Sri Lanka participated in exploratory factor analysis (EFA). Responses from 90 different undergraduate nursing students (3rd and 4th academic years) (University of Peradeniya [n=42] and Eastern University, Sri Lanka [n=48]) were employed for confirmatory factor analysis (CFA).

Setting

This cross-sectional study was conducted in 2 steps. The first step involved the development of the NSR-FC. The second step was to investigate the psychometric properties of the NSR-FC.

Step 1: Developing a scale for measuring nursing students’ readiness for the flipped classroom

The NSR-FC was drafted after reviewing the literature pertaining to learner readiness. The following existing inventories were used to generate items for the NSR-FC: E-Learning Readiness [7]; Online Learning Readiness Scale [6], and the ICT literacy scale [8]. However, none of the existing inventories had been developed specifically for nursing education. Therefore, 18 items (Q1–3, Q5–6, Q15–17, Q21–24, Q26, Q28–29, and Q33–35) were generated specifically to assess FC readiness by our research team based on our experience in teaching and learning in clinical and nursing education. Furthermore, 19 items were modified from the existing inventories and included in the NSR-FC. As a result, 37 preliminary items were included in the NSR-FC. A 5-point Likert scale was used, with responses ranging from 1 (strongly disagree) to 5 (strongly agree) as scale response anchors (Supplement 1).

Step 2: Exploring the psychometric properties of the NSR-FC

Content validation of the NSR-FC

Content validation was done to assess the level of representativeness, relevance, understandability, and completeness of the NSR-FC. Five Sri Lankan content experts including 3 senior academics in health profession education and 2 academics in nursing participated and individually evaluated the degree of item significance for nursing students’ readiness to engage in the FC. The content validity indices (CVI) at the average scale-level (S-CVI/average) and item-level (I-CVI) were calculated using descriptive statistics. The I-CVI for each item on the NSR-FC was computed as the number of experts giving a rating of 1 (not relevant) to 4 (highly relevant) divided by the total number of experts who responded to the item. An I-CVI score of 0.78 or higher was considered to be adequate [9]. The S-CVI/average was calculated as the sum of the I-CVIs divided by the total number of items. An S-CVI/average score of 0.90 or above was considered as acceptable [9].

Construct validation of the NSR-FC

EFA and CFA were conducted for the investigation of construct validity of the NSR-FC. In total, 265 undergraduate nursing students participated in EFA. The Kaiser–Meyer–Olkin (KMO) test and the Bartlett test of sphericity were used to test the adequacy of the study’s sample and the suitability of orthogonal factor rotation [10]. Principal component analysis (PCA) was used with varimax rotation to identify the factors of the NSR-FC items that related to the corresponding variables. A parallel analysis was performed to confirm the extracted factors by comparing eigenvalues obtained from the raw data sets and randomly generated parallel dataset [11]. The factors that received eigenvalues higher than those from the corresponding datasets were included for further analysis [11]. The cut-off extraction value of factor loading was determined as 0.4 or above [12].

The degree of model fit was assessed through CFA, which utilized responses from 90 different undergraduate nursing students (Dataset 2). Four goodness-of-fit indices were calculated to estimate the global fit of the NSR-FC: the comparative fit index (CFI, with a threshold of >0.90); the root mean square error of approximation (RMSEA; a value of <0.05 was considered to indicate close fit, and a value between 0.05–0.08 to indicate reasonable fit); standardized root mean square residual (SRMR, with a threshold ≤0.08); and the chi-square statistic and its ratio to degrees of freedom (χ²/degrees of freedom [df], with a threshold of <5) [6,13].

Internal consistency of the NSR-FC

Internal consistency is commonly used to indicate the degree of reliability of a self-reporting questionnaire. The internal consistency of the NSR-FC was measured by computing the Cronbach α coefficient. A Cronbach α of 0.70 or higher was considered to indicate acceptable internal consistency, meaning that the observed score variance is reliable when compared with the true score variance [14].

Study size

In accordance with the recommendation made by Lynn [15], 5 content experts were recruited for content validation. In total, 365 students participated in the study. Cattell [16] in 1978 recommended 3 to 6 samples per variable when conducting EFA. Therefore, we randomly selected students in their 1st and 2nd academic years ([35*6]<[n=265]) for EFA and the reliability test. The remaining sample (n=90) was used for CFA, which was sufficient to produce good agreement between the sample and population solutions (K value=0.92) [17].

Statistical methods

Descriptive statistics were employed to calculate CVI measures using Microsoft Excel (Microsoft Corp., Redmond, WA, USA). PCA was applied using IBM SPSS ver. 26.0 (IBM Corp., Armonk, NY, USA) to explore the factor structure. Cronbach α coefficients were computed using IBM SPSS ver. 26.0 (IBM Corp.). The CFA goodness-of-fit indices were calculated using AMOS ver. 26.0 (IBM Corp.) [8,10].

Results

Content validation

At the item level, 91.9% of the NSR-FC’s items (n=34) had an I-CVI greater than or equal to 0.90. Two items of the NSR-FC (E1 & E2) received an I-CVI less than 0.78, namely “I can discipline myself to follow flipped learning” (I-CVI=0.75) and “I am committed to using flipped learning” (I-CVI=0.45). In addition, reviewers reported that E1 and E2 duplicated existing items, so these 2 items were excluded from the questionnaire. The S-CVI/average of the questionnaire achieved an acceptable level of 0.94 (Supplement 1).

Exploratory factor analysis of the NSR-FC

The NSR-FC was explored using PCA to determine the optimal model that best represented the data. The KMO value of the NSR-FC was 0.873, suggesting that the samples were adequate for PCA. The Bartlett test of sphericity further affirmed the suitability of the data for PCA with orthogonal factor rotation (χ²=4,717.18, P<0.001). The PCA revealed that NSR-FC could be reduced to 5 factors, namely technological readiness, environmental readiness, personal readiness, pedagogical readiness, and interpersonal readiness. Parallel analysis confirmed the 5 factors in the NSR-FC. The percentage of variance explained by the rotated factor matrices ranged from 4.29% to 25.42% per factor, with the 5 factors explaining 55.25% of the overall variance. Factor loading after the rotation of each item is shown in (Table 1, Dataset 1).

Accordingly, technological readiness (factor 1) included 12 items (Q6–16 and Q18). The factor had a cumulative eigenvalue of 8.89 and accounted for 25.42% of the total variance. Environmental readiness (factor 2) comprised 5 items (Q20–21 and Q23–25), had a cumulative eigenvalue of 4.38, and accounted for 12.51% of the total variance. Personal readiness (factor 3) contained 6 items (Q1–4, Q17, and Q19). The factor had a cumulative eigenvalue of 2.81, and accounted for 8.02% of the total variance. Pedagogical readiness (factor 4) encompassed 7 items (Q22, Q26–29, and Q34–35), had a cumulative eigenvalue of 1.75, and accounted for 5.01% of the total variance. Interpersonal readiness (factor 5) included 4 items (Q30–33) and had a cumulative eigenvalue of 1.50, accounting for 4.29% of the total variance. Item Q5 obtained a factor extraction value of 0.25, which was deemed to be low; this item was therefore excluded from the NSR-FC.

Internal consistency of the NSR-FC

The Cronbach α values of the NSR-FC’s factors ranged from 0.76 to 0.93, and the Cronbach α for the scale as a whole was 0.90, indicating excellent reliability. The intraclass correlation coefficients (ICCs) for the NSR-FC’s factors showed satisfactory results for each subscale (ICCs ranging from 0.76 to 0.92; P<0.01), and the overall ICC was 0.90, exhibiting an acceptable level of reliability. The corresponding data are presented in (Table 2, Dataset 1).

Confirmatory factor analysis of the NSR-FC

Model 1 of the CFA denotes the baseline model, with 34 items as identified by PCA. In reference to model 1, the goodness-of-fit indices did not achieve an acceptable level, except for χ²/df. Therefore, model 2 was created by excluding 5 items (Q7, 10, 16, 22, and 33) that showed loading values less than 0.4; however, the goodness-of-fit values of model 2 were still below the cut-off level (Table 3, Dataset 2).

In addition, all items in factor 5 did not obtain an acceptable loading value. Therefore, it was decided to exclude factor 5 from model 3. Model 3 was found to be the model with the best fit in the current study (Fig. 1). According to SRMR and an adjunct discrepancy-based fit index (χ²/df), model 3 achieved an acceptable level of fit (SRMR=0.08 and χ²/df=1.57). In reference to RMSEA, model 3 demonstrated reasonable fit (RMSEA=0.08). The CFI of model 3 (0.87) came close to meeting the cut-off level (Table 3).

Discussion

Interpretation

Student readiness is recognized as a valuable factor for determining pedagogical effectiveness [4]. Therefore, the present study investigated the development and psychometric properties of a scale used to measure nursing students’ readiness for the FC. More specifically, the study explored the construct validity and reliability of the NSR-FC.

The method used to construct the NSR-FC was similar to the procedure used for the development of student readiness scales in other disciplines, and included proposing constructs, item generation, analysis of the content, item reduction, and validation of the newly developed instrument [6,7]. PCA revealed a potential 5-factor structure for the NSR-FC. However, CFA was only able to confirm a 4-factor model as determined by an inspection of the fit indices. The 4 factors that best fit the data with respect to the NSR-FC included technological readiness, environmental readiness, personal readiness, and pedagogical readiness.

The 4 best-fitting factors of the NSR-FC were comparable with existing inventories used to measure students’ readiness for blended learning in school-level education. Technological readiness is a common subset, identified as technology self-efficacy in many other studies [6,18]. Technological readiness denotes an individual’s willingness to leverage novel technologies for carrying out tasks [19]. Items relating to environmental readiness in the NSR-FC were seen to load together with the technological factor in the E-Learning Readiness Scale [7]. Personal readiness was recognized as a factor in the NSR-FC. Since FC is a student-centred pedagogical approach, students play a significant role in teaching-learning activities. Therefore, it is necessary to estimate students’ individual willingness to engage in the FC. Personal readiness was identified as “learner control,” with some variation, in the Online Learning Readiness Scale [6]. Pedagogical readiness in the NSR-FC describes students’ willingness to embrace learning content through FC pedagogy. A few items in the pedagogical readiness factor were correlated with the subscale of “online communication self-efficacy” in the Online Learning Readiness Scale [6] and the “content factor” in the E-Learning Readiness Scale [7].

In the reliability assessment, the NSR-FC demonstrated acceptable internal consistency. All 4 factors of the NSR-FC generated Cronbach α and ICC values greater than 0.7, confirming internal consistency within the domains and the ability of the NSR-FC to generate reproducible results. Therefore, the 4-factor model of NSR-FC is a valid and reliable tool that can be used to measure students’ readiness.

Limitation/generalizability

This instrument can be used by nursing educators and curriculum planners to evaluate the effectiveness of FC pedagogy. However, the results of factor analysis may have been sample-specific, and the generalizability of these results is subject to the similarity of respondents to the sample. Since the study was conducted in Sri Lanka, the NSR-FC could be used to assess nursing students’ readiness for the FC in the context of South Asia. Moreover, the CFI value of the NSR-FC did not meet the acceptable level. This may have been due to the limited sample size and is acknowledged as a limitation of the study. Thus, future research is encouraged to perform CFA of the NSR-FC with a larger sample size. Lastly, the study was conducted through a self-reported questionnaire survey, which may have resulted in reporting biases such as social desirability bias [20]. Qualitative research will likely be instructive in terms of enabling a deeper understanding of the phenomena underlying nursing students’ readiness for the FC.

Conclusion

These findings indicate that the NSR-FC is an acceptable instrument for measuring nursing students’ readiness for the FC in reference to its construct validity and reliability within the Sri Lankan nursing education context. The 4 readiness subscales were found to be technological readiness, environmental readiness, personal readiness, and pedagogical readiness. This finding may provide a good platform and frame of reference for nursing curriculum researchers and educational designers regarding the necessity of assessing students’ readiness for gaining actual educational achievements through the FC.

Acknowledgments

Supplementary materials

Fig. 1.

Model 3 with factor loadings for the 20-item Nursing Students’ Readiness for Flipped Classroom.

References

• 1. Oh J, Kim SJ, Kim S, Vasuki R. Evaluation of the effects of flipped learning of a nursing informatics course. J Nurs Educ 2017;56:477-483. https://doi.org/10.3928/01484834-20170712-06 ArticlePubMed
• 2. Dehghanzadeh S, Jafaraghaee F. Comparing the effects of traditional lecture and flipped classroom on nursing students’ critical thinking disposition: a quasi-experimental study. Nurse Educ Today 2018;71:151-156. https://doi.org/10.1016/j.nedt.2018.09.027 ArticlePubMed
• 3. Oh J, Kim SJ, Kim S, Kang KA, Kan J, Bartlett R. Development and evaluation of flipped learning using film clips within a nursing informatics course. Jpn J Nurs Sci 2019;16:385-395. https://doi.org/10.1111/jjns.12245 ArticlePubMed
• 4. Thorndike EL. The fundamentals of learning. New York (NY): Teachers College, Columbia University; 1932.
• 5. Yildiz Durak H. Flipped learning readiness in teaching programming in middle schools: modelling its relation to various variables. J Comput Assist Learn 2018;34(6):939-959. https://doi.org/10.1111/jcal.12302 Article
• 6. Hung ML, Chou C, Chen CH, Own ZY. Learner readiness for online learning: scale development and student perceptions. Comput Educ 2010;55:1080-1090. https://doi.org/10.1016/j.compedu.2010.05.004 Article
• 7. Akaslan D, Law EL. Measuring student e-learning readiness: a case about the subject of electricity in higher education institutions in turkey. In: Leung H, Popescu E, Cao Y, Lau RW, Nejdl W, editors. Advances in Web-Based Learning: ICWL 2011. Berlin: Springer; 2011. https://doi.org/10.1007/978-3-642-25813-8_22
• 8. Lau WW, Yuen AH. Developing and validating of a perceived ICT literacy scale for junior secondary school students: pedagogical and educational contributions. Comput Educ 2014;78:1-9. https://doi.org/10.1016/j.compedu.2014.04.016 Article
• 9. Halek M, Holle D, Bartholomeyczik S. Development and evaluation of the content validity, practicability and feasibility of the innovative dementia-oriented assessment system for challenging behaviour in residents with dementia. BMC Health Serv Res 2017;17:554. https://doi.org/10.1186/s12913-017-2469-8 ArticlePubMedPMC
• 10. Wang Y, Wu L, Zhou H, Xu J, Dong G. Development and validation of a self-reported questionnaire for measuring Internet search dependence. Front Public Health 2016;4:274. https://doi.org/10.3389/fpubh.2016.00274 ArticlePubMedPMC
• 11. O’Connor BP. SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behav Res Methods Instrum Comput 2000;32:396-402. https://doi.org/10.3758/bf03200807 ArticlePubMed
• 12. Osborne JW. Best practices in quantitative methods [Internet]. Thousand Oaks (CA): SAGE Publications Inc.; 2008 [cited 2020 Oct 13]. Available from: https://methods.sagepub.com/book/best-practices-in-quantitative-methods
• 13. Alyami M, Henning M, Krageloh CU, Alyami H. Psychometric evaluation of the Arabic version of the fear of COVID-19 scale. Int J Ment Health Addict 2020 May 16 [Epub]. https://doi.org/10.1007/s11469-020-00316-x Article
• 14. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297-334. https://doi.org/10.1007/BF02310555 Article
• 15. Lynn MR. Determination and quantification of content validity. Nurs Res 1986;35:382-385. https://doi.org/10.1097/00006199-198611000-00017 ArticlePubMed
• 16. Cattell RB. The scientific use of factor analysis in behavioral and life sciences. New York (NY): Plenum Press; 1978.
• 17. Mundfrom DJ, Shaw DG, Ke TL. Minimum sample size recommendations for conducting factor analyses. Int J Test 2005;5:159-168. https://doi.org/10.1207/s15327574ijt0502_4 Article
• 18. Chou CL, Hung ML, Tsai CW, Chang YC. Developing and validating a scale for measuring teachers’ readiness for flipped classrooms in junior high schools. Br J Educ Technol 2020;51:1420-1435. https://doi.org/10.1111/bjet.12895 Article
• 19. Parasuraman A. Technology Readiness Index (TRI): a multiple-item scale to measure readiness to embrace new technologies. J Serv Res 2000;2:307-320. https://doi.org/10.1177/109467050024001 Article
• 20. Tullis T, Albert B. Self-reported metrics. In: Tullis T, Albert B, editors. Measuring the user experience: collecting, analyzing, and presenting usability metrics. 2nd ed. Boston (MA): Morgan Kaufmann; 2013. 121-161.

Figure & Data

References

Citations

Citations to this article as recorded by  

• Design and validation of a preliminary instrument to contextualize interactions through information technologies of health professionals
  José Fidencio López Luna, Eddie Nahúm Armendáriz Mireles, Marco Aurelio Nuño Maganda, Hiram Herrera Rivas, Rubén Machucho Cadena, Jorge Arturo Hernández Almazán
  Health Informatics Journal.2024;[Epub]     CrossRef
• AI readiness scale for teachers: Development and validation
  Mehmet Ramazanoglu, Tayfun Akın
  Education and Information Technologies.2024;[Epub]     CrossRef
• Content validity of the Constructivist Learning in Higher Education Settings (CLHES) scale in the context of the flipped classroom in higher education
  Turki Mesfer Alqahtani, Farrah Dina Yusop, Siti Hajar Halili
  Humanities and Social Sciences Communications.2023;[Epub]     CrossRef
• The intensivist's assessment of gastrointestinal function: A pilot study
  Varsha M. Asrani, Colin McArthur, Ian Bissett, John A. Windsor
  Australian Critical Care.2022; 35(6): 636.     CrossRef
• Psychometric evidence of a perception scale about covid-19 vaccination process in Peruvian dentists: a preliminary validation
  César F. Cayo-Rojas, Nancy Córdova-Limaylla, Gissela Briceño-Vergel, Marysela Ladera-Castañeda, Hernán Cachay-Criado, Carlos López-Gurreonero, Alberto Cornejo-Pinto, Luis Cervantes-Ganoza
  BMC Health Services Research.2022;[Epub]     CrossRef
• Implementation of a Web-Based Educational Intervention for Promoting Flipped Classroom Pedagogy: A Mixed-Methods Study
  Punithalingam Youhasan, Mataroria P. Lyndon, Yan Chen, Marcus A. Henning
  Medical Science Educator.2022; 33(1): 91.     CrossRef
• Assess the feasibility of flipped classroom pedagogy in undergraduate nursing education in Sri Lanka: A mixed-methods study
  Punithalingam Youhasan, Yan Chen, Mataroria Lyndon, Marcus A. Henning, Gwo-Jen Hwang
  PLOS ONE.2021; 16(11): e0259003.     CrossRef
• Newly appointed medical faculty members’ self-evaluation of their educational roles at the Catholic University of Korea College of Medicine in 2020 and 2021: a cross-sectional survey-based study
  Sun Kim, A Ra Cho, Chul Woon Chung
  Journal of Educational Evaluation for Health Professions.2021; 18: 28.     CrossRef