fidoptMSD Liver
Contrast-enhanced CT of the abdomen of a patient with a primary liver tumour or liver metastases, from the LiTS challenge's clinical sites, with the liver and its tumours segmented (Medical Segmentation Decathlon, Task03 Liver).
10 test casesabdomenimage CT
Read before comparing (9)
- Source. Task03 is the data of LiTS 2017: contrast-enhanced abdominal CT of patients with primary liver tumours or metastases, before and after therapy, from seven sites, some with metal artefacts. Neither provider says which scan comes from which site, nor that a patient gives one scan only. The decathlon calls the scans portal venous; LiTS says only contrast-enhanced.
- Truth. LiTS's labels: a radiologist with more than 3 years in oncologic imaging drew the scans slice by slice in ITK-SNAP, three further blinded readers verified them, and very small, uncertain lesion-like structures were left unlabelled. In the LiTS paper a radiologist who redrew the tumours of 15 scans agrees at a median Dice of 0.70, and two radiologists' corrections left the labels at 0.95.
- Labels. Label 1 is the liver without its tumours: TotalSegmentator's liver overlaps it at a Dice of 0.836 on liver_100 (537 ml of tumours) but both labels together at 0.971. The tumours hold 2.0 to 537 ml on the test cases (median 122 ml) in 1 to 23 pieces. Test case liver_87 and development case liver_119 have no tumour label.
- Images. The decathlon's files as delivered, in Hounsfield units and not clipped: liver_16 and liver_4 go down to -2048, liver_87 up to 14591 and liver_117 up to 17486. On the test cases pixels are 0.56 to 0.95 mm and slices 0.7 to 5 mm apart.
- CTORG. CT-ORG's 131 dedicated CTs are LiTS's, and CTORG holds 20 of them, left out here: its test cases liver_0, 2, 6, 7, 8, 10, 12, 13, 15 and 21 and its development cases liver_24, 25, 48, 63, 69, 75, 97, 99, 114 and 122, each matched by slice count, pixel size and a slice voxel for voxel.
- 3D-IRCADb-01. liver_28 to liver_47 are left out: their headers give 1 mm in every direction, the scanner's geometry lost, and IRCAD's page puts them under CC BY-NC-ND 4.0. CTORG leaves out the same 20.
- Other datasets. The headers of all 201 volumes were compared with every image built for CRLMCT, LiverMetsCT, HCCTACE, LiverLesionsCT, MSD_HepaticVessel, LiverPhasesCT, LiverDIRQA and LiverUSCT: none matches in slice count and pixel size but LiverPhasesCT's P0058 and liver_41, which differ by 1279 HU on the slice compared.
- Exposure. The test cases are training cases of LiTS and of the decathlon, public with their labels, and CT-ORG's other scans, not matched, include some of them with public liver labels: a network trained on LiTS, on the decathlon's liver task or on CT-ORG has seen them.
- Outside numbers. None: the decathlon and LiTS rank on hidden test sets. There the 19 entries' mean Dice ran from 0.29 to 0.95 on the liver (median 0.94) and from 0.15 to 0.74 on the tumours (median 0.54; nnU-Net 0.74): methods part on the tumours. The surface Dice here is at 7 mm for both labels, the clinicians' tolerance and the loosest of the ten tasks.
| Accuracy | Cost | |||||||
|---|---|---|---|---|---|---|---|---|
| # | Method | Dice | Surface Dice | HD95 mm | Time s | RAM GB | GPU GB | Images |
| 11–3 | KonfAI segmentation app · TotalSegmentator (total), KonfAI appGPUapp1 of 2 structures | 0.921 ±0.039 | 0.949 ±0.025 | 10.69 ±4.56 | 22 | 2.9 | 6.5 | |
| 21–3 | KonfAI segmentation app · TotalSegmentator (total), the authors' packageGPUapp1 of 2 structures | 0.920 ±0.039 | 0.949 ±0.025 | 10.73 ±4.65 | 19.3 | 5.8 | ||
| 31–3 | CT-FM whole-body segmentationGPUapp1 of 2 structures1 failed | 0.831 ±0.212 | 0.811 ±0.211 | 17.51 ±9.60 | 220 | 15.1 | 6.9 | |
| 44–5 | MONAI wholeBody CTGPUapp1 of 2 structures8 failed | 0.185 ±0.280 | 0.192 ±0.290 | 7.18 ±32.44 | 20 | 4.2 | 8.2 | |
| 54–5 | KonfAI segmentation app · TotalSegmentator (liver_vessels), the authors' packageGPUapp1 of 2 structures | 0.591 ±0.249 | 0.689 ±0.273 | 34.26 ±24.21 | 42 | 24.6 | 3.5 | |
| 6 | KonfAI segmentation app · MRSegmentator, KonfAI appGPUapp1 of 2 structures2 failed | 0.006 ±0.007 | 0.021 ±0.026 | 150.48 ±20.85 | 57 | 12.3 | 8.8 | |
The order averages each method's ranks on Dice, Surface Dice, HD95 mm, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 10 cases. Two neighbours are tied when a paired Wilcoxon test on the shared cases does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Workstation.
1 result not ranked: an older run, cases still missing, or a failure on every case
| Accuracy | Cost | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| # | Method | Dice | Surface Dice | HD95 mm | Time s | RAM GB | GPU GB | Machine | Images |
| MONAI wholeBody CTGPUapp1 of 2 structures1 failedolder run | 0.824 ±0.210 | 0.846 ±0.214 | 10.67 ±4.55 | 29 | 3.6 | 16.3 | Laptop | ||
Dice: how much the organ contours overlap, 1 is perfect. Surface Dice: the share of the contours' surfaces within a tolerance of each other, 1 is perfect. HD95: distance between the contours' surfaces.
Case by case
Click a score to look at that result: the image, the truth in green and the method's map in red. Underlined scores have their image; for the others you see the case without the result.
About the data
The first test caseClick to look at every case.
- Test cases
- 10 test cases drawn with seed 2026 among the 81 training cases outside development, and 10 development cases. The pool is the decathlon's 131 training cases less the 20 scans CTORG holds and the 20 scans of 3D-IRCADb-01 (liver_28 to liver_47). The labels of the decathlon's 70 test scans were never released.
- Truth
- contours by experts the decathlon's two labels, which are LiTS's: 1 the liver without its tumours and 2 the tumours, any neoplastic lesion, primary or metastatic, drawn by hand slice by slice in ITK-SNAP by a radiologist with more than 3 years of experience in oncologic imaging and verified by three further readers
- Source
- medicaldecathlon.com
arxiv.org/abs/1902.09063
doi.org/10.1038/s41467-022-30695-9
msd-for-monai.s3-us-west-2.amazonaws.com/Task03_Liver.tar
arxiv.org/abs/1901.04056
doi.org/10.1016/j.media.2022.102680
competitions.codalab.org/competitions/17094
www.ircad.fr/research/data-sets/liver-segmentation-3d-ircadb-01 - Licence
- CC BY-SA 4.0
10 cases for tuning, never tested
liver_53, liver_72, liver_68, liver_119, liver_94, liver_88, liver_117, liver_4, liver_58, liver_11