CLEAR item#20

“Details of operators performing segmentation. State how many readers performed the segmentation, as well as their experience. In the case of multiple readers, describe how the final form of segmentation is achieved (e.g., the consensus of readers, intersection of segmentations, independent segmentation for further reproducibility analysis, sequential refinements from numerous expert raters until convergence), which is particularly important for the training data because the segmentation process on the test data should be as close to the clinical practice as possible, that is, the segmentation of a single reader.” [1] (from the article by Kocak et al.; licensed under CC BY 4.0)

Reporting examples for CLEAR item#20

Example#1. “Supervised manual delineation of all primary tumors was performed by S.M. and B.P. (both 3 years of experience) with visual inspection of delineation by senior head and neck radiologists (P.G. or F.P. with 11 and 25 years of experience).” [2] (from the article by Mes et al.; licensed under CC BY 4.0)

Example#2. “Segmentation was performed under standardized lighting conditions by consensus reading of two experienced observers (G.K. and S.Z.) and quality-controlled by an abdominal radiologist with more than 10 years of experience in pancreatic MRI” [3] (from the article by Kaissis et al.; licensed under CC BY 4.0)

Example#3. “Manual lesion segmentation was carried out by one expert radiologist with more than 10 years of experience in breast MRI on the co-registered images using ImageJ.” [4] (from the article by D’Amico et al.; licensed under CC BY 4.0)

Example#4. “VOI was drawn on each image slice that contained the optic nerve tissue by two radiologists with more than 5 years of experience in diagnosing DON independently. The radiologists were unaware of the results of diagnoses (TAO with or without DON).” [5] (from the article by Wu et al.; licensed under CC BY 4.0)

Explanation and elaboration of CLEAR item#20

In Item#20, the disclosure of operator experience and the process of achieving final segmentation in medical imaging is emphasized. Example#1 highlights a two-tiered approach: initial segmentation by operators with three years of experience, followed by review and validation from senior radiologists with 11 to 25 years of experience [2]. This setup combines initial assessments with expert validation to ensure accurate and reliable segmentation. Example#2 is notable for clearly outlining the process to reach the final segmentation. It involves a consensus-based approach, where two experienced observers perform the segmentation and a senior abdominal radiologist conducts a quality check [3]. This explicit detailing of the consensus process for finalizing segmentation is exemplary, as it provides transparency and underscores the collaboration among experts to ensure the precision and dependability of the segmentation. Such clarity in reporting is vital for aligning with clinical practices and reinforcing the credibility of the segmentation outcomes. However, despite their positive connotation, according to recent evidence related to radiomics, it should also be noted that such consensus approaches or corrective techniques described in Example#1 and Example#2 require further reproducibility analyses because the radiomic features still have significant reproducibility problems [6]. In the Example#3, the segmentation was performed by one reader and in Example#4, the task was performed by two independent radiologists for the purpose of inter-observer reproducibility analysis.

References

  1. Kocak B, Baessler B, Bakas S, et al (2023) CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII. Insights Imaging 14:75. https://doi.org/10.1186/s13244-023-01415-8
  2. Mes SW, van Velden FHP, Peltenburg B, et al (2020) Outcome prediction of head and neck squamous cell carcinoma by MRI radiomic signatures. Eur Radiol 30:6311–6321. https://doi.org/10.1007/s00330-020-06962-y
  3. Kaissis G, Ziegelmayer S, Lohöfer F, et al (2019) A machine learning model for the prediction of survival and tumor subtype in pancreatic ductal adenocarcinoma from preoperative diffusion-weighted imaging. Eur Radiol Exp 3:41. https://doi.org/10.1186/s41747-019-0119-0
  4. D’Amico NC, Grossi E, Valbusa G, et al (2020) A machine learning approach for differentiating malignant from benign enhancing foci on breast MRI. Eur Radiol Exp 4:5. https://doi.org/10.1186/s41747-019-0131-4
  5. Wu H, Luo B, Zhao Y, et al (2022) Radiomics analysis of the optic nerve for detecting dysthyroid optic neuropathy, based on water-fat imaging. Insights Imaging 13:154. https://doi.org/10.1186/s13244-022-01292-7
  6. Kocak B, Yardimci AH, Nazli MA, et al (2023) REliability of consensus-based segMentatIoN in raDiomic feature reproducibility (REMIND): A word of caution. Eur J Radiol 165:110893. https://doi.org/10.1016/j.ejrad.2023.110893

Back Next