Introduction

Background/rationale

Reporting guidelines can be defined as “a checklist, flow diagram, or structured text to guide authors in reporting a specific type of research, developed using explicit methodology” [1]. Reporting guidelines have been developed for various study designs. A journal encourages authors to follow these guidelines because they help researchers explain their studies sufficiently to be evaluated by editors, reviewers, readers, and other researchers [2]. The Journal of Educational Evaluation for Health Professionals (JEEHP) is one of the academic journals that has recommended the use of reporting guidelines in writing manuscripts. Since 2021, JEEHP has adopted a journal policy that requires reporting guidelines to be described in the Methods section. Compliance with reporting guidelines is essential for enhancing transparency in journals. Therefore, the policies of journals related to reporting guidelines should not only have a simple declarative meaning but also mandate that authors accurately determine the study design of the manuscript, select appropriate reporting guidelines, and write the manuscript according to the reporting guidelines.

Studies related to reporting guidelines in academic journals have investigated the adoption rate of reporting guideline policies in academic journals [3] and the ratio of declaring reporting guidelines in the articles [4]. However, since these studies focused only on specific reporting guidelines such as CONSORT (Consolidated Standards of Reporting Trials), they do not provide a comprehensive overview of issues related to reporting guidelines. Especially not many academic journals mandate the declaration of reporting guidelines, so the case of JEEHP can be considered unique. In this regard, it would be meaningful to review whether the study design and the corresponding recommendation reporting guidelines have been adequately indicated in papers published in JEEHP.

Objectives

The purpose of this study was to suggest the study design and the corresponding reporting guidelines in research articles, review articles, and brief reports published in JEEHP from January 2021 to September 2022. Specifically, the following were investigated: the proportiom of articles describing the study design; the proportion of articles describing reporting guidelines; the types of study design and corresponding reporting guidelines; and, a suggestion of a more suitable study design based on the study design algorithm for medical literature on interventions (DAMI), systematic reviews & other review types, and epidemiological studies overview.

Methods

Ethics statement

It is not a human study but a literature-based study. Therefore, neither approval by the institutional review board nor obtainment of informed consent is required.

Setting

Articles in JEEHP published from January 2021 to September 2022 were searched for the presence of study design and corresponding reporting guidelines.

Subjects

Research articles, review articles, and brief reports were included among the papers published during the period. Editorials, educational/faculty development materials, and technical reports were excluded.

Variables

Study design and reporting guidelines were variables.

Measurement

General characteristics of the included articles

The general characteristics assessed were as follows: (1) type of research (research article, review, brief report, case report); (2) description of study design: whether a description of the study design was present in the Methods section; and (3) type of study design: study design classified by the author (S.Y.K.) referring to research design classification systems, such as DAMI (a study design algorithm for medical literature on interventions) [5], systematic reviews & other review types (https://guides.temple.edu/c.php?g=78618&p=4156607), and the epidemiological studies overview (https://lo.unisa.edu.au/mod/book/view.php?id=646428).

Suggestion of a more suitable study design description

It was examined whether the study design classification described in the Methods section and the author’s (S.Y.K.) research classification were consistent.

Availability of reporting guidelines and amore suitable selection of reporting guidelines

It was examined whether the reporting guidelines indicated by the author were enough. The reporting guidelines were searched in EQUATOR (The EQUATOR Network and UK EQUATOR Centre, Oxford, UK) to determine whether they were available according to the most up-to-date classification of study designs. It was also examined whether the reporting guidelines indicated by the author were suitable.

Raw analysis data of 45 articles are available from Dataset 1.

Bias

There was no selection bias.

Study size

Sample size estimation is unnecessary since all target articles were included in the analysis.

Statistical methods

Descriptive statistics were used.

Results

General characteristics of the included articles

Among the 59 papers published in JEEHP from 2021 to September 2022, the number of research articles, review articles, and brief reports was 45. Among the 45 publications, the most common type of publication was original articles (n=30, 66.7%), followed by reviews and brief reports. Out of 45 studies, 44 described the study design (97.8%), but only half indicated this information in the title or abstract. In the actual study design of the 45 papers re-classified by the expert (S.Y.K.), the most common study designs were descriptive cross-sectional surveys (n=14, 31.1%), followed by before-and-after studies (n=9, 20.0%), diagnostic accuracy tests (n=5, 11.1%), and non-randomized trials (n=4, 8.9%). Eighteen articles indicated the reporting guidelines in the Methods section (Table 1).

Suggestion of a more suitable study design description

There were 25 articles (55.6%) in which the indicated study design matched a study design endowed by the expert (S.Y.K.). A descriptive cross-sectional survey was the most common study design that can be classified again. The authors of JEEHP described descriptive cross-sectional surveys using other study design terms (content analysis study, analysis study, comprehensive study, network analysis, mixed study; n=5, 25%). In other cases, the actual study design was a before-and-after study, but the authors described it as a comprehensive observational study, methodological study, or cross-sectional study (n=5, 25%). In addition, studies that could be classified as diagnostic studies were described as explicit studies, simulation studies, or other terms (n=4, 20%) (Table 2).

Availability of reporting guidelines

Eighteen (40.0%) of the 45 articles indicated reporting guidelines (Table 1). The largest percentage was for diagnostic studies (STARD [Standards for the Reporting of Diagnostic Accuracy Studies]; n=5, 27.8%), followed by non-randomized trials (TREND [Transparent Reporting of Evaluations with Nonrandomized Designs]; n=4, 22.2%), systematic reviews (Preferred Reporting Items for Systematic Reviews and Meta-Analyses [PRISMA]; n=3, 16.7%), scoping reviews (PRISMA-ScR; n=2, 11.1%), and analytic observation studies (Strengthening the Reporting of Observational Studies in Epidemiology [STROBE]; n=2, 11.1%) in descending order (Table 3).

However, 27 of the 45 papers (60%) did not indicate corresponding reporting guidelines. The largest percentage of these were descriptive cross-sectional surveys (n=14, 51.9%), followed by before-and-after studies (n=9, 33.3%), and narrative reviews (n=3, 11.1%) (Table 4).

Suggestion of reclassification of reporting guideline

Of the 18 reporting guidelines indicated by authors, 8 (44.4%) were well-matched. Among the 10 articles in which a re-selection of reporting guidelines was suggested, the most frequent was the application of STROBE to descriptive studies (n=5, 50%). In addition, there were cases where STROBE was applied to before-and-after studies (n=4, 40.0%) and a randomized controlled trial (n=1, 10.0%) (Table 5).

Discussion

Key results

Out of 45 research articles in JEEHP from January 2021 to 2022, 44 described the study designs (97.8%), but only 25 (55.6%) were considered well-matched study designs. Other 19 study design descriptions that could be re-classified mainly occurred in studies in which the actual study designs were before-and-after studies, diagnostic research, and non-randomized trials. Of the 18 reporting guidelines indicated by authors, 8 (44.4%) were well-matched. Other descriptions of reporting guidelines that can be reselected included the application of STROBE to non-applicable research (descriptive studies, before-and-after studies, and randomized controlled trials).

Interpretation

Out of 44 study design descriptions, 41.3% recorded in the Methods sections of papers published in JEEHP needs to be, and 60.0% of the described reporting guidelines also need to be re-classified. If a study design does not match the content, the reporting guidelines will not match. Therefore, these 2 results are correlated.

Comparison with previous studies

No previous reports examined the match of study designs with content and reporting guidelines simultaneously. However, one previous study examined the appropriateness of the reporting guidelines described in papers. Innocenti et al. [4] stated that reporting guidelines were in 17.5% of rehabilitation research articles, and among these studies, 48.6% (95% confidence interval, 32%–65.1%) used study designs appropriately. In the present study, 18 (40.0%) out of 45 studies declared the use of reporting guidelines, which is a higher rate than in previous studies, but the proportion of well-matched use was 44.4% out of 18 reporting guideline descriptions. No previous paper has suggested reasons why studies did not declare reporting guidelines. According to the results of this paper, the problem is that reporting guidelines sometimes cannot be declared because many study designs do not have corresponding reporting guidelines. Only 40.0% of the studies analyzed herein could select the reporting guidelines. Therefore, if reporting guidelines are developed for descriptive cross-sectional surveys or before-and-after studies, the rate of declaration of reporting guidelines may increase. The study design description in 20 out of 45 articles needs to be re-classified in the present study. Previous studies have examined whether the study design described in the Methods section was appropriate. One study investigated the appropriateness of the classification of research designs published in 3 dermatology journals and found that 26.5% of observational studies had inappropriately described research designs [6]. In 4 Asian dermatology journals in Science Citation Index-Expanded, 72 publications (40.7%) of the 177 papers in 2018 declared their research design. Of those, 23 articles (32.0%) demonstrated differences between the author-reported and the actual study designs [7]. The results of this study can be seen as similar to or slightly higher than those of previous studies.

Limitations/generalizability/suggestions

There were limitations to this study. First, judgments of the match of study design with content and the corresponding reporting guidelines involved some subjectivity, although the author has expertise in classifying study designs. Second, the small number of target papers is another limitation. The analysis was conducted only on papers published after 2021, and a more general conclusion would have been possible if the period had been extended.

Conclusion

According to the above result, education and training on both study design and reporting guidelines are necessary. In addition, it may be necessary to establish a more realistic reporting guideline policy that reflects the actual development of reporting guidelines. The proportion of papers published in JEEHP after 2021 that declared reporting guidelines was not high, and the match of the description of the study design with content and reporting guidelines was also not high. Education and training on study design and reporting guidelines for researchers are needed. Reporting guideline policies should also be implemented considering the current development status of reporting guidelines.

Article information

Authors’ contributions

All the work was done by Soo Young Kim.

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Funding

None.

Data availability

Data files are available from Harvard Dataverse: https://doi.org/10.7910/DVN/TYWWZ2

Dataset 1. Data file containing the study design and guidelines of articles published from 2021 to September 2022 in the Journal of Educational Evaluation for Health Professions.

jeehp-19-36-dataset.xlsx

Acknowledgments

Supplementary materials

Article information

Editor’s note

There are many study designs in the Equator network (https://www.equator-network.org/) and the United States National Library of Medicine Research Reporting Guidelines and Initiatives (https://www.nlm.nih.gov/services/research_report_guide.html). I recommended descriptions that I believed were reasonable for each manuscript and corresponding reporting guideline in the Methods section. I should confirm whether my recommended description is suitable. The most critical point is that there are no suitable reporting guidelines for descriptive studies and before-and-after studies. For a while, I recommended that authors describe their work as cross-sectional observational studies instead of descriptive studies and suggested that the authors write the manuscript according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement. I frequently suggested that authors re-describe the study design and corresponding reporting guidelines. Therefore, those descriptions in JEEHP do not solely reflect the authors’ opinions. It is time to evaluate my trial attempt to recommend that authors provide a written description of the study design and corresponding reporting guidelines. I invited Dr. Soo Young Kim, an expert in this field, to check whether the study designs and corresponding reporting guidelines were suitable. He is the author of the “study design algorithm for medical literature on interventions” (DAMI) classification. In the present study, he indicated that some study designs and reporting guidelines should be re-classified as more suitable ones. I will follow his opinion. However, since there are no suitable reporting guidelines for descriptive studies, I will continue using the STROBE statement for these studies. Furthermore, I eagerly anticipate the development of reporting guidelines for descriptive studies by an expert group.

References

• 1. Kim SY. Reporting guidelines. Korean J Fam Med 2009;30:62. https://doi.org/10.4082/kjfm.2009.30.1.62 Article
• 2. International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing and publication of scholarly work in medical journals [Internet]. Washington (DC): International Committee of Medical Journal Editors; 2022 [cited 2022 Oct 21]. Available from: https://www.icmje.org/icmje-recommendations.pdf
• 3. Zuniga-Hernandez JA, Dorsey-Trevino EG, Gonzalez-Gonzalez JG, Brito JP, Montori VM, Rodriguez-Gutierrez R. Endorsement of reporting guidelines and study registration by endocrine and internal medicine journals: meta-epidemiological study. BMJ Open 2019;9:e031259. https://doi.org/10.1136/bmjopen-2019-031259 ArticlePubMedPMC
• 4. Innocenti T, Salvioli S, Giagio S, Feller D, Cartabellotta N, Chiarotto A. Declaration of use and appropriate use of reporting guidelines in high-impact rehabilitation journals is limited: a meta-research study. J Clin Epidemiol 2021;131:43-50. https://doi.org/10.1016/j.jclinepi.2020.11.010 ArticlePubMed
• 5. Seo HJ, Kim SY, Lee YJ, Jang BH, Park JE, Sheen SS, Hahn SK. A newly developed tool for classifying study designs in systematic reviews of interventions and exposures showed substantial reliability and validity. J Clin Epidemiol 2016;70:200-205. https://doi.org/10.1016/j.jclinepi.2015.09.013 ArticlePubMed
• 6. Ohn J, Eun SJ, Kim DY, Park HS, Cho S, Yoon HS. Misclassification of study designs in the dermatology literature. J Am Acad Dermatol 2018;79:315-319. https://doi.org/10.1016/j.jaad.2017.10.049 ArticlePubMed
• 7. Choi S, Yoon HS. Pervasive misclassification and misconception of study designs in Asian dermatology journals listed in Science Citation Index-Expanded. Ann Dermatol 2020;32:383-387. https://doi.org/10.5021/ad.2020.32.5.383 ArticlePubMedPMC

Figure & Data

References

Citations

Citations to this article as recorded by  

• Issues in the 3rd year of the COVID-19 pandemic, including computer-based testing, study design, ChatGPT, journal metrics, and appreciation to reviewers
  Sun Huh
  Journal of Educational Evaluation for Health Professions.2023; 20: 5.     CrossRef
• A comprehensive perspective on the interaction between gut microbiota and COVID-19 vaccines
  Ming Hong, Tin Lan, Qiuxia Li, Binfei Li, Yong Yuan, Feng Xu, Weijia Wang
  Gut Microbes.2023;[Epub]     CrossRef
• Why do editors of local nursing society journals strive to have their journals included in MEDLINE? A case study of the Korean Journal of Women Health Nursing
  Sun Huh
  Korean Journal of Women Health Nursing.2023; 29(3): 147.     CrossRef