fidoptMSD Pancreas
Portal venous abdominal CT of a patient before resection of a pancreatic mass at Memorial Sloan Kettering Cancer Center, with the pancreas and the mass segmented (Medical Segmentation Decathlon, Task07 Pancreas).
Read before comparing (9)
- Pool. The decathlon's Task07: 420 portal venous CTs of patients operated for a pancreatic mass (intraductal mucinous neoplasm, pancreatic neuroendocrine tumour or ductal adenocarcinoma) at Memorial Sloan Kettering. The archive holds 281 training scans with labels (the site says 282) and 139 test scans; the cases here are training cases.
- Truth. One expert abdominal radiologist drew both labels by hand on every slice in the Scout application; no second reading is reported. Label 1 is the gland without its mass, label 2 the mass, cyst or tumour: a method that segments the whole pancreas loses where the mass is. Every map of the 53 cases holds 0, 1 and 2 on its image's grid.
- Sizes. The mass holds 0.9 to 14 ml on the ten test cases drawn first (median 2.4 ml) and 1.5 to 92 ml on the other eighteen; the gland 28 to 173 ml and 20 to 185 ml.
- Fragments. On 15 of the pool's 280 labels, 12 of them numbered 365 to 415, the pancreas falls in more than ten pieces. Test cases pancreas_387 and pancreas_414 are two of them, with 16 and 20 pieces, all but two or three of a few voxels.
- TotalSegmentator. On pancreas_006 its pancreas, about half as large as labels 1 and 2 together, overlaps label 1 at a Dice of 0.58 (HD95 10.0 mm, surface Dice 0.81 at 5 mm) and both labels at 0.68, and holds 92% of the mass. On pancreas_055 it overlaps label 1 at 0.72.
- Images. The decathlon's NIfTI files as delivered: Hounsfield units with values below -1024 set to -1024, the voxels transposed without resampling to the closest RAS orientation.
- Overlap. Two of the challenge's test scans are training scans byte for byte (pancreas_027 is pancreas_032, pancreas_007 is pancreas_228, a development case). No other dataset of the corpus holds these scans: PancreasCT8 is NIH's Pancreas-CT, and pancreas_029 and pancreas_074, which share the slice count and pixel size of AbdomenDIRQA's case20 and CRLM-CT-1174, hold no slice of them. FLARE22 holds 50 of the training scans with its own labels, the four drawn here left out: its 20 drawn cases stay out of this pool if the dataset grows.
- Public labels. The test cases are public training cases with their labels: a network trained on the decathlon's pancreas task, as the challenge's entries were, has seen them.
- Room. On the hidden test set the 19 entries' mean Dice ran from 0.48 to 0.79 on the pancreas (median 0.69; nnU-Net 0.79) and from 0.01 to 0.52 on the mass (median 0.21; nnU-Net 0.52). The surface Dice is at 5 mm, the error the challenge's clinicians accepted. No outside number compares: the challenge ranks on its hidden test set.
| Accuracy | Cost | |||||||
|---|---|---|---|---|---|---|---|---|
| # | Method | Dice | Surface Dice | HD95 mm | Time s | RAM GB | GPU GB | Images |
| 11–2 | KonfAI segmentation app · TotalSegmentator (total), KonfAI appGPUapp1 of 2 structures | 0.758 ±0.031 | 0.913 ±0.029 | 10.24 ±3.52 | 26 | 3.6 | 6.2 | |
| 21–2 | KonfAI segmentation app · TotalSegmentator (total), the authors' packageGPUapp1 of 2 structures | 0.758 ±0.030 | 0.910 ±0.030 | 10.59 ±3.63 | 17.9 | 3.8 | ||
| 3 | MONAI wholeBody CTGPUapp1 of 2 structures6 failed | 0.562 ±0.121 | 0.698 ±0.146 | 10.37 ±3.00 | 17 | 3.9 | 7.1 | |
| 4 | CT-FM whole-body segmentationGPUapp1 of 2 structures8 failed | 0.353 ±0.115 | 0.490 ±0.144 | 25.16 ±12.41 | 22 | 8.5 | 8.3 | |
| 5 | KonfAI segmentation app · MRSegmentator, KonfAI appGPUapp1 of 2 structures1 failed | 0.032 ±0.033 | 0.039 ±0.040 | 91.96 ±17.57 | 22 | 5.7 | 6.8 | |
The order averages each method's ranks on Dice, Surface Dice, HD95 mm, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 28 cases. Two neighbours are tied when a paired Wilcoxon test on the shared cases does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Workstation.
Dice: how much the organ contours overlap, 1 is perfect. Surface Dice: the share of the contours' surfaces within a tolerance of each other, 1 is perfect. HD95: distance between the contours' surfaces.
Case by case
Click a score to look at that result: the image, the truth in green and the method's map in red. Underlined scores have their image; for the others you see the case without the result.
About the data
- Test cases
- 28 test cases drawn with seed 2026 among the 255 training cases outside development, out of a pool of 280, and 25 development cases. pancreas_291, a copy of pancreas_290, is left out. The labels of the decathlon's 139 test scans were never released.
- Truth
- contours by experts the decathlon's two labels, drawn by hand on every slice by an expert abdominal radiologist in the Scout application: 1 the pancreatic parenchyma, the mass left out of it, and 2 the mass, cyst or tumour
- Source
- medicaldecathlon.com
arxiv.org/abs/1902.09063
doi.org/10.1038/s41467-022-30695-9
msd-for-monai.s3-us-west-2.amazonaws.com/Task07_Pancreas.tar - Licence
- CC BY-SA 4.0
25 cases for tuning, never tested
pancreas_228, pancreas_194, pancreas_310, pancreas_323, pancreas_074, pancreas_327, pancreas_211, pancreas_331, pancreas_045, pancreas_395, pancreas_111, pancreas_344, pancreas_283, pancreas_098, pancreas_181, pancreas_129, pancreas_095, pancreas_404, pancreas_279, pancreas_369, pancreas_166, pancreas_058, pancreas_114, pancreas_147, pancreas_376