INTRODUCTION

The concept of Evidence-Based Medicine (EBM) has been vigorously followed in medical research and implemented in clinical practice since the early 1990s, exerting a profound impact on advancement of all medicine disciplines ^[1]. Nevertheless, fully integrating EBM into clinical practice remains a formidable challenge ^[2]. Richard Horton, the current editor-in-chief of The Lancet, has remarked, “Much of the scientific literature, perhaps half, may simply be untrue” ^[3]. Similarly, it had been estimated that the majority of published research findings are likely to be false, and up to 30% of the most influential initial medical research papers may subsequently be discovered to be incorrect or exaggerated ^[4]. 

A fundamental EBM is the currently available evidence derived from clinical research presented in the published literature ^[5]. However, the quality of publications varies significantly, which complicates the process of selecting published information for decision-making in both medical research and clinical practice ^[6].

Study design is a decisive factor influencing the quality of literature. It plays a crucial role in determining the levels of evidence (LOE) and guiding clinical practice ^[2]. Each type of study design has its own advantages and limitations, which can greatly affect the conduct, outcomes and applicability of the study. Despite the publication of several guidelines aimed at enhancing the clarity and accuracy of study design reporting, such as the STROBE Statement (Strengthening the Reporting of Observational Studies in Epidemiology) ^[7], many publications still reported incorrect study designs ^[8]. Previous bibliometric research has showed that mislabeling and unclear reporting of SD were prevalent issues in various specialties, including ophthalmology ^[9,10], orthopedics ^[11], and dermatology ^[12]. For example, a literature survey revealed that mislabeled case-control and cohort studies could occur in 17.7% (201 of 1135) of articles published in international ophthalmology journals from 1990 to 2020 ^[9]. Such incorrect or inadequate labeling undoubtedly makes it more difficult for readers, especially beginners, to understand research methodologies. Moreover, these issues can result in inappropriate or even incorrect sorting and indexing of publications, which would severely disrupt the use of updated information in the literature for the practice of EBM.

The scientific literature, including ophthalmology as well as all other medical disciplines, is expanding at a rapid pace, making it challenging to ensure high-quality in all publications. In recent years, ophthalmology in China has witnessed remarkable progress and has emerged as a significant force in the global landscape of ophthalmic research and clinical practice ^[13]. The leading ophthalmic journals in China have been steadily improving in terms of quality, quantity and influence. As a result, the quality of research published in these journals may influence the decision-making processes of clinicians in their clinical practices and scientists in their research endeavors. A previous study described and analyzed the study designs in Chinese ophthalmic journals in 2019. It revealed that 142/350 (40.6%) articles did not specify a particular study design, and 36/211 articles (17.06%) were incorrectly labeled ^[10]. It remains unknown whether there has been an improvement in the understanding of study design in recent years.

Therefore, this study aimed to examine the study designs of clinical research articles published in three major Chinese ophthalmic journals over the past three years. The objectives were to determine the rate of mislabeling and relevant factors, and to assess the impact based the estimation of LOE.

METHODS

Identification of studies

This is a bibliometric study. Based on the 2023 edition of the overview of Chinese core journals and the list of key journals in China’s technology sector, we identified the top three general ophthalmic journals as ranked by China Journal Citation Reports (CJCR): the Chinese Journal of Ophthalmology, Chinese Journal of Experimental Ophthalmology, and Recent Advances in Ophthalmology. We searched the Wanfang Database for all articles published in these three journals from 2021 to 2023. The completeness and accuracy of the collected data were verified against the journal's official directories. In cases where data were unavailable or discrepancies arose between our assessment and the reported data, we referred to the official website as the definitive reference.

We included articles of original clinical research and meta-analysis based on clinical research from the three ophthalmology journals published from 2021 to 2023. Clinical research was defined as research involving human subjects with the aim of understanding or managing human diseases and other health conditions, including: 1) patient-oriented research; 2) epidemiological and behavioral research; 3) outcomes research and health services research.

Articles published in sections not related to clinical ophthalmology, such as Experimental Science, Editorial, Discussion of Standard, Image Essence, Education, Focus, Editorial Note, Experience Sharing, Management Model, and Technical Exchange, as well as articles focusing on medical humanities, health economics, pharmacology, and other non-clinical research topics, were excluded. Narrative reviews were also excluded.

Study review, definition and standardization

For the selected articles, we recorded fundamental information, including author details, sub-specialty, specific diseases, sample source. The sub-specialties were classified with reference to the American Academy of Ophthalmology's Basic and Clinical Science Course (BCSC).

During the review process, the author details and journal information were concealed from the investigators. First, the original classification was determined based on the labels assigned to the study design in the original text. If the original text did not specify the study design or merely labeled it as a “prospective” or “retrospective” study, it was defined as “unlabeled”.

Subsequently, three investigators (HZS, RFZ, and ZHX) independently reviewed the abstracts and full texts of all enrolled studies according to Essential Concepts in Clinical Research ^[14] to determine the study design (Supplementary table 2). The studies were classified into primary studies and secondary studies (meta-analysis) (Figure 1). Articles in the primary study category were further classified into observational studies, interventional studies and diagnostic tests. The observational studies were further divided into analytic studies (cohort studies, case-control studies, cross-sectional studies, and ecological studies) and descriptive studies (case series, pedigree studies, and case reports). The interventional studies were further classified into experimental studies (randomized controlled trials [RCTs]) and quasi-experimental studies (non-RCT and before-after studies). Eventually, 12 study designs were identified. In the case where articles had more than one study design (n=3), the one with a higher LOE was considered the primary design. If there were discrepancies among the investigators, a senior investigator (HXW) as the final arbiter to determine a definitive classification.

Finally, we assessed the impact of mislabeling by analyzing the actual LOE according to the study design we determined, and comparing it with the reported LOE based on the study design stated by the author in the original text. To standardize the LOE, we graded the evidence of the original and re-evaluated studies in accordance with the Joanna Briggs Institute's (JBI) evidence hierarchy introduced in 2014 (Supplementary Table 1) (https://jbi.global/sites/default/files/2020-07/Supporting_Doc_JBI_Levels_of_Evidence.pdf). This approach ensured that our analysis was consistent with established criteria for evaluating research quality. Studies that were uncommon or not included in the reference scale (diagnostic tests and ecological studies) were excluded. Specifically, we defined pedigree studies as having the same LOE as case series because they were similar in research method and temporal direction.

Statistical analysis

Descriptive statistics were employed to characterize the mislabeling. Qualitative data were presented in terms of frequency and percentage. A Sankey diagram was generated using Microsoft Power BI Desktop (Version 2.128.952.0) to visually illustrate the comparison and transformation between the original classification and reclassification of the SD. The Chi-square test and Fisher's exact test were used to examine the relationship between mislabeling and various factors. Cramer's V coefficient was applied to assess the strength of this relationship. According to established published evaluation criteria, the strength of the relationship was classified as follows: no correlation (Cramer’s V coefficient less than 0.1), weak correlation (between 0.1 and 0.3), moderate correlation (between 0.3 and 0.5), and strong correlation (between 0.5 and 1.0).

Statistical analysis was carried out using SPSS software version 23.0 (SPSS, Inc., Chicago, IL). A p-value less than 0.05 was deemed statistically significant.

RESULTS

In the initial screening phase, a total of 1,870 articles from the 3 journals were selected. Subsequently, 1115 articles were excluded. After reviewing the abstracts and full-text of the remaining articles, 11 non-clinical studies and studies that did not meet our study’s inclusion criteria were further excluded. Ultimately, 744 articles were included for analysis (Table 1 and Figure 2).

The independent classifications conducted by the three researchers showed a high level of consistency, with a Kappa value of 0.78, P<0.001. However, inter-researcher discrepancies were identified in 72 (9.68%) of the articles. Among the 744 articles, many studies focused on t therapy, accounting for 25.4% (n=189). The most common sub-specialties, in descending order, were Retina/Vitreous, Refractive Management/Intervention, and Cornea/External Disease. Most of the articles centered on the specific disease that are frequently encountered in clinical settings, such as ametropia, cataract, diabetic retinopathy and glaucoma.

Study design & level of evidence

Among the 744 articles included, only 531 (71.4%) clearly stated their study design. According to the original classification, case series was the most frequently labeled study design (n=122). However, after reclassification, cross-sectional studies emerged as the most common study design, with 194 articles(n=194), followed by the case series with 153 articles (n=153) (Figure 3). For the 531 articles that clearly reported a specific study design, 59 (11.1%) were mislabeled. Before-after studies were frequently mislabeled, accounting for 24 articles (72.7% of mislabeled articles in this context), followed by RCTs, with 5 articles (13.2% of mislabeled articles).

A total of 213 (28.6%) articles did not label the specific study design, even though a description of the methodology was provided. Among these articles with unlabeled study designs, 190 were observational studies, 14 were quasi-experimental studies, and 9 were experimental studies.

Table 2 illustrates the changes in LOE after the reclassification of SD. After reclassification, 45 (6.0%) articles were level I, 58 (7.8%) were level II, 148 (19.9%) were level III and 483 (64.9%) were level IV. Of the 521 studies that clearly mentioned the study design, 20 (3.8%) over-reported their LOE and 39 (7.5%) under-reported their LOE.

Relevant factors associated with mislabeling

Table 3 showed the association between study design mislabeling and factors such as journal, literature topic, sub-specialty, and originally labeled study design, with no significant annual difference observed (P=0.584). Among three journals, Recent Advances in Ophthalmology had the highest mislabeling rate (n=16, 23.9%). There was a high rate of mislabeling in articles focusing on prognosis (n=14, 23.7%) and therapy (n=27, 21.4%). Frequent mislabeling was also found in the specialties of Pediatric Ophthalmology/Strabismus (n=4, 30.8%),Refractive Management/Intervention (n=16, 16.8%) and Cataract/Anterior Segment (n=11, 16.2%). Specifically, the originally labeled study design showed a moderate correlated with mislabeling, as indicated by a Cramer's V value of 0.408. Articles originally labeled as non-RCT, case-control study, and case series were the most frequently mislabeled, with the mislabeling rate of 41.4%, 29.6%, and 23.0%, respectively.

DISCUSSION

This bibliometric study was conducted to review the clinical research articles published in three major Chinese ophthalmology journals and investigate the accuracy of study design reporting. We found that 11.1% of the articles incorrectly labeled their study designs, and 28.6% did not specify a particular study design. These findings indicate notable issues with the accurate reporting of study designs in these three Chinese ophthalmic journals. Compared with a previous bibliometric study, which showed that in 2019, 40.2% of articles in relevant journals did no state their study designs and 17.06% mislabeled them ^[10] our findings showed a modest improvement over the past few years.

In this study, we found that the majority of the published articles in ophthalmology journals were observational studies (n=633, 85.1%), a proportion similar to that in dermatology (80%) ^[12]. This study also revealed that over the past three years, the primary focus of major Chinese ophthalmology journals has been on therapy, which aligns with the findings of a previous bibliometric analysis of Chinese ophthalmology journals conducted in 2019 ^[10]. In contrast, a separate study examining 92 ophthalmic articles published in the Journal of the American Medical Association (JAMA) found that research on etiology and risk factors was the most prevalent between 2000 and 2014, accounting for 32.6% of the total published articles ^[15].

The study design plays a crucial role in determining its level of evidence (LOE), which reflects the risk of bias and robustness of findings. Generally, the more rigorous the study design, the higher the level of evidence it provides, leading to greater confidence in the results. 

In this study, we observed a tendency among mislabeled articles to report the LOE as lower than the actual level. In contrast, previous research in three international dermatology journals showed that the LOE were more likely to be over-reported due to mislabeling ^[12]. The opposite conclusion may stem from the fact that different journals have preferences for various types of study designs, and that different authors may exhibit varying degrees of reporting tendencies. For instance, authors submitting their work to international journals might be more confident and definite in their reporting compared to those who choose local journals. Furthermore, there is a widespread belief that studies with higher levels of evidence are more likely to be accepted by international journals ^[16]. This preferential treatment not only enhances the authors' reputation but also potentially brings financial benefits. Accumulated wealth and an enhanced reputation, in turn, confer significant advantages in accessing academic resources and securing research grants. A study shows that high-quality ophthalmic journals publish more RCTs, which belongs to high LOE ^[17]. For journals, selecting articles with a high LOE may correlates with an increased citation rate. This, in turn, can elevate the journal's standing within the academic community, attract higher-quality submissions, enhance its academic discourse power, and augment its economic benefits. Based on the above reasons, the occurrence of this phenomenon is well founded, regardless of whether such outcomes are intentional.

In summary, the results of this study suggested a considerable degree of mislabeling of study designs in publications of three Chinese ophthalmic journals despite availability of guidelines and criteria designed to assist the authors. Improving in quality of ophthalmology literature requires collaborative efforts involving authors, publishers and reviewers.

Correction notice

None

Acknowledgements

None

Author Contributions

(I) Conception and design: Hongzi Shen, Ruifeng Zhang, Zihao Xu, Mingzhi Zhang,  Hongxi Wang

(II) Administrative support: Zijing Huang , Geng Wang,Kunliang Qiu, Chi-Pui Pang, Mingzhi Zhang, Hongxi Wang

(III) Provision of study materials or patients: Hongzi Shen, Ruifeng Zhang, Zihao Xu, Ting Zeng, Xin Tan, Gehan Ai

(IV) Collection and assembly of data:Hongzi Shen, Ruifeng Zhang, Zihao Xu

(V) Data analysis and interpretation: Hongzi Shen, Ruifeng Zhang, Pi Guo 

(VI) Manuscript writing: All authors

(VII) Final approval of manuscript: All authors

Funding

This work was supported by 2023 Guangdong Provincial Clinical Teaching Base of General Higher Medical Colleges Undergraduate Teaching Reform Research Project (2023JD131), Key research platforms and projects of universities in Guangdong Province (2023KCXTD013). 

Conflict of Interests

None of the authors has any conflicts of interest to disclose. All authors have declared in the completed the ICMJE uniform disclosure form.

Patient consent for publication

None

Ethics approval and consent to participate

None

Data availability statement

None

Open access

This is an Open Access article distributed in accordance with the Creative Commons AttributionNonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license).