Item #2

“Eligibility criteria that describe a representative study population” [1] (licensed under CC BY)

Explanation

“Whether inclusion and exclusion criteria are explicitly defined. These should lead to a representative study sample that matches the general population of interest for the study aim.” [1] (licensed under CC BY)

Positive examples from the literature

Example #1: “Patients who had undergone BCa surgical excision were verified pathologically and were treated at two different institutions between January 2014 and September 2019. […] The inclusion criteria were as follows: (1) For BCa, the availability of pathological data; (2) for lesion evaluation, complete CT images of all four phases of the CTU examination; and (3) CT examinations performed within 30 days before surgery. The exclusion criteria were: (1) Poor image quality of the CT or preoperative CTU; (2) lesions with hard-to-define edges; (3) if the patient received immunotherapy or chemotherapy before the CT; and (4) missing clinico-pathological data on the patient.” [2] (licensed under CC BY)

Example #2: “All consecutive patients with a single renal mass referred to the Urology Department of Pisa University Hospital and underwent CECT before surgical treatment between January 2016 and December 2020 were considered eligible for this retrospective study. The inclusion criteria were as follows: having been diagnosed at CECT with a T1a renal mass (i.e., tumor confined to the kidney, < 4 cm) within 1 month before surgery, confirmed as ccRCC or RO at histopathological examination. Patients with missing unenhanced phases and either one between arterial or venous phases on CECT examinations, or poor image quality (e.g., motion artifact), were excluded.” [3] (licensed under CC BY-NC)

Also see Figure 1.

Fig1 Figure 1. “Flowchart of patient selection.” [3] (licensed under CC BY-NC)

Example #3: “We retrospectively enrolled patients affected by LACC, staged IB2 to IVA from International Federation of Gynecology and Obstetrics (FIGO) 2018, treated with NACRT followed by radical hysterectomy plus pelvic lymphadenectomy after 6–8 weeks. […] Inclusion criteria were as follows: histological confirmed invasive carcinoma of the cervix, FIGO stage from IB2 to IVA and absence of distant metastasis. Patients with incomplete documentation, younger than 18 years, treated with palliative intent and did not undergo surgery were excluded.” [4] (licensed under CC BY)

Also see Figure 2.

Fig2

Figure 2. “Flowchart of our population” [4] (licensed under CC BY)

Example #4: “A thorough assessment of consecutive patients who had undergone surgical resection at a single tertiary cancer hospital was conducted within the timeframe spanning from July 22, 2014, to June 15, 2018. The criteria for inclusion of patients were listed below: (1) pathology-confirmed newly diagnosed rectal cancer; (2) clinical stage. II to III (cT3-4M0 and/or positive nodal status) at pre-nCRT MRI scan; (3) designated for nCRT prior to surgical resection; (4) underwent total mesorectal excision with postoperative pathological assessment after completion of nCRT; and (5) pre- and post-nCRT MRI images were available. The criteria for exclusion were listed below: (1) prior anti-cancer therapy; (2) poor image quality due to stent insertion; and (3) a time duration exceeding 12 weeks between post-nCRT MRI and surgery.” [5] (licensed under CC BY-NC-ND)

Also see Figure 3.

Fig3

Figure 3. “Flowchart summarizes patient selection and allocation to the training and validation sets” [5] (licensed under CC BY-NC-ND)

Hypothetical negative examples

Example #5: We included patients with brain tumors who underwent MRI at our institution between 2015 and 2022. Patients with missing data or poor imaging quality were excluded.

Example #6: We retrospectively analyzed lung cancer patients who underwent CT imaging in our radiology department. Only cases with solid nodules were included, while patients with small nodules (<2 cm) or those with poor contrast enhancement were excluded. In total, 100 cases were analyzed.

Example #7: We retrospectively analyzed 50 male patients diagnosed with prostate cancer at our hospital between January 2020 and December 2021. Only patients aged between 50 and 55 years, with a PSA level between 5-7 ng/mL, and who underwent MRI on a 3T scanner were included. Patients with a history of prior cancer, comorbidities, or incomplete data were excluded.

Example #8: Patients diagnosed with pneumonia between 2018 and 2023 were included. Patients with confirmed bacterial pneumonia were excluded. Additionally, cases with mild pneumonia that did not require hospitalization were removed from the dataset.

Importance of the item

Inclusion and exclusion criteria are critical for defining a representative study sample aligned with the study’s aim [6]. Inclusion criteria outline the key characteristics of the target population required to address the research question, while exclusion criteria identify variables in eligible participants that may hinder study validity [7, 8]. Properly designed criteria are pivotal in enhancing external validity and applicability of results. Common errors include redundancies, such as listing the same variable as inclusion and exclusion criteria or using unrelated variables. Effective eligibility criteria may encompass clinical, demographic, and diagnostic traits, yet they are frequently underreported, undermining the transparency and relevance of study findings [9].

Specifics about the positive examples

All the examples provided include the relevant information necessary for a positive scoring in METRICS, with slight differences. Example #1 outlines eligibility criteria effectively, listing both inclusion and exclusion criteria without redundancy. Furthermore, Example #2 incorporates a basic flowchart in the results section, from eligible patients up to the included cohort, further divided into two groups according to the study objective. Example #3, a multicentric study, similarly includes a detailed flowchart illustrating all patients excluded at various stages, specifying the reasons for each exclusion. Example #4 also stands out by providing a comprehensive flowchart that traces the process from eligible patients to the final included cohort, detailing every step of enrollment comprehensively.

Specifics about the negative examples

Example #5 fails to specify tumor type, stage, or diagnostic confirmation methods. Additionally, terms like “missing data” and “poor imaging quality” are vague and undefined, making it unclear what qualifies as “poor quality imaging.” Example #6 introduces a selection bias by excluding all cases with poor contrast enhancement, potentially leading to a dataset that does not reflect real-world diagnostic challenges and is not representative of clinical practice. Example #7 restricts the sample to only patients aged 50-55 years with a specific PSA range, thereby excluding a large segment of real-world patients and compromising generalizability. Additionally, many prostate cancer patients have comorbidities, and excluding them further introduces bias and reduces representativeness. Example #8 weakens the study’s validity by excluding bacterial pneumonia, one of the most common forms of pneumonia, making the eligibility criteria non-representative and non-generalizable. Furthermore, excluding mild pneumonia skews the dataset towards severe cases, meaning AI models trained on this data may fail in real-world settings, where mild cases are more frequent.

Recommendations for appropriate scoring

For accurate scoring on this criterion, studies must be distinguished by a representative sample that aligns with the general population, according to the study’s objective. This necessitates that raters carefully evaluate the eligibility as well as the inclusion and exclusion criteria.

Ambiguously defined or inadequately detailed criteria may hinder the representativeness of the sample and significantly undermine the reproducibility of the study. Such deficiencies warrant consideration of a negative score due to their potential impact on the validity and generalizability of the findings.

References

  1. Kocak B, Akinci D’Antonoli T, Mercaldo N, et al (2024) METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII. Insights Imaging 15:8. https://doi.org/10.1186/s13244-023-01572-w
  2. Ren J, Gu H, Zhang N, Chen W (2023) Preoperative CT-based radiomics for diagnosing muscle invasion of bladder cancer. Egypt J Radiol Nucl Med 54:131. https://doi.org/10.1186/s43055-023-01044-7
  3. Francischello R, Fanni SC, Chiellini M, et al (2024) Radiomics-based machine learning role in differential diagnosis between small renal oncocytoma and clear cells carcinoma on contrast-enhanced CT: A pilot study. European Journal of Radiology Open 13:100604. https://doi.org/10.1016/j.ejro.2024.100604
  4. Autorino R, Gui B, Panza G, et al (2022) Radiomics-based prediction of two-year clinical outcome in locally advanced cervical cancer patients undergoing neoadjuvant chemoradiotherapy. Radiol med 127:498–506. https://doi.org/10.1007/s11547-022-01482-9
  5. Liu J, Liu K, Cao F, et al (2024) MRI-based radiomic nomogram for predicting disease-free survival in patients with locally advanced rectal cancer. Abdom Radiol. https://doi.org/10.1007/s00261-024-04710-0
  6. Vandenbroucke JP, Von Elm E, Altman DG, et al (2007) Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration. PLoS Med 4:e297. https://doi.org/10.1371/journal.pmed.0040297
  7. Cohen JF, Korevaar DA, Altman DG, et al (2016) STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open 6:e012799. https://doi.org/10.1136/bmjopen-2016-012799
  8. Kocak B, Baessler B, Bakas S, et al (2023) CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII. Insights Imaging 14:75. https://doi.org/10.1186/s13244-023-01415-8
  9. Patino CM, Ferreira JC (2018) Inclusion and exclusion criteria in research studies: definitions and why they matter. J bras pneumol 44:84–84. https://doi.org/10.1590/s1806-37562018000000088

Previous: Item #1 Next: Item #3