fidopt Interactive view
Abdomen CT, segmentation

FLARE22

Portal venous abdominal CT of a patient before resection of a pancreatic mass at Memorial Sloan Kettering (the decathlon's Task07 Pancreas), with 13 abdominal organs segmented for the FLARE 2022 challenge.

10 test casesabdomenimage CTresearch only
Look at the images
Read before comparing (10)
  • Pool. The deposit's 50 labelled CTs, the challenge's labelled training set, are decathlon Task07 training scans: each equals one decathlon file in byte size, NIfTI header and first 64 KB (FLARE22_Tr_0001 is pancreas_001, FLARE22_Tr_0050 pancreas_313). The matcher finds the decathlon's three known copies (pancreas_027 and 032, 007 and 228, 290 and 291).
  • Truth. Twelve organs start from a network's proposal; the paper gives no agreement between readers for these cases. Graded expert with that doubt, as AMOS. The pancreas label holds the mass and is a revision of the decathlon's: a Dice of 0.988, 0.973 and 0.955 with the decathlon's labels 1 and 2 together on FLARE22_Tr_0001, _0021 and _0050.
  • Labels. Every map of the 50 holds all 13 labels, as 16-bit integers on its CT's grid; 16 of them carry only a qform, with the CT's geometry.
  • Scoring. The challenge scored the aorta, the inferior vena cava and the oesophagus only on the slices their truth covers; the benchmark scores them on the whole image.
  • Images. The decathlon's NIfTI files as delivered: Hounsfield units as 32-bit floats, values below -1024 set to -1024, slices 2.5 mm apart on 47 cases and 4 or 5 mm on 3.
  • Cases. The ten test CTs hold 71 to 110 slices of 0.64 to 0.90 mm pixels. FLARE22_Tr_0026's slices lie 4 mm apart and its stomach holds oral contrast (a median of 193 HU); FLARE22_Tr_0014's pancreas label, 148 mL at a median of -21 HU, is mostly a cystic mass.
  • TotalSegmentator. Version 2.18 (task total) on test case FLARE22_Tr_0001 finds the 13 organs at a mean Dice of 0.900 and a mean surface Dice of 0.954 at the challenge's tolerances, the duodenum lowest (0.768). Its aorta adds a piece 7 cm in front of the descending aorta on the top five slices, which the truth leaves out (HD95 55 mm).
  • Overlap. MSD_Pancreas holds four of these scans (pancreas_006, 029 and 055 in its test set, pancreas_095 in development): they are held out. Matched by slice count, matrix and pixel size against the 13,216 built images of the corpus, the 50 find those four and three others: AbdomenDIRQA's case20 (pancreas_029, ruled out by MSD_Pancreas), TotalSegMR's s0363 (an MR) and PANORAMA's 101178_00001, which holds no slice of FLARE22_Tr_0019 (a mean difference of 94 HU or more; 0 HU for FLARE22_Tr_0003 against pancreas_006). PANORAMA leaves the decathlon's patients out.
  • Public labels. The cases are public training cases of FLARE22, AbdomenCT-1K and the decathlon, with their labels: a network trained on any of them has seen them.
  • Room. On the challenge's hidden internal validation set the top three teams, trained on these 50 cases and 2,000 unlabelled ones, reached a median Dice of 0.886 to 0.894 over the 13 organs (0.916 for the algorithms with the best Dice). No outside number compares: the challenge ranks on its hidden sets.
AccuracyCost
#MethodDiceSurface DiceHD95 mmTime sRAM GBGPU GBImages
11–2KonfAI segmentation app · TotalSegmentator (total), the authors' packageGPUapp0.891 ±0.0100.951 ±0.0105.70 ±1.4515.83.6
21–2KonfAI segmentation app · TotalSegmentator (total), KonfAI appGPUapp0.890 ±0.0110.948 ±0.0115.36 ±1.18253.46.3
3MONAI wholeBody CTGPUapp0.868 ±0.0170.926 ±0.0166.49 ±1.42173.97.6
44–5CT-FM whole-body segmentationGPUapp5 failed0.395 ±0.2980.416 ±0.31512.22 ±4.40147.69.1
54–5KonfAI segmentation app · MRSegmentator, KonfAI appGPUapp0.011 ±0.0050.016 ±0.007149.66 ±12.136.66.1

The order averages each method's ranks on Dice, Surface Dice, HD95 mm, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 10 cases. Two neighbours are tied when a paired Wilcoxon test on the shared cases does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Workstation.

Dice: how much the organ contours overlap, 1 is perfect. Surface Dice: the share of the contours' surfaces within a tolerance of each other, 1 is perfect. HD95: distance between the contours' surfaces.

Case by case

Click a score to look at that result: the image, the truth in green and the method's map in red. Underlined scores have their image; for the others you see the case without the result.

About the data

Test cases
10 test cases drawn with seed 2026 among the 46 cases of the deposit's 50 that no other dataset holds, and 10 development cases, the last of the same seeded order. draw.held_out lists the four that MSD_Pancreas holds. The challenge's tuning, validation and test labels are not public.
Truth
contours by experts FLARE22's 13 organs, in the challenge's numbering: the decathlon's pancreas and mass, drawn by hand by an expert abdominal radiologist and revised, and twelve organs from AbdomenCT-1K, a network's proposal corrected by junior annotators (one to five years of experience) and verified by a senior radiologist (over ten years), the cases a cross-validated network scored low checked again
Licence
CC BY 4.0 (the Zenodo record's licence field); the record's description gives the organ labels from AbdomenCT-1K for research purposes only
10 cases for tuning, never tested

FLARE22_Tr_0023, FLARE22_Tr_0039, FLARE22_Tr_0022, FLARE22_Tr_0015, FLARE22_Tr_0002, FLARE22_Tr_0031, FLARE22_Tr_0035, FLARE22_Tr_0006, FLARE22_Tr_0040, FLARE22_Tr_0005