PENGWIN CT

Pelvic CT of a patient with a pelvic fracture before reduction surgery, from six medical centres in China, with the sacrum and the two hip bones labelled (PENGWIN 2024 challenge, task 1).

10 test casespelvisimage CT

Look at the images

Read before comparing (6)

Pool. 150 patients aged 16 to 94, scanned before surgery on seven scanner models, without significant artefact, cropped to the pelvis; the challenge's 100 training CTs are public. The CTs are the provider's files unchanged, in Hounsfield units, some reading -2048 or below outside the scanner's field of view.
Labels. The dataset labels each fragment of a broken bone. prepare.py merges each bone's fragments into sacrum (1), hip_left (2) and hip_right (3), as the challenge's anatomy scores do: a fragment has no name that holds across patients. What is scored is each bone of a broken pelvis, fracture lines and displaced pieces included. The ten test CTs hold 4 to 9 fragments each.
Truth. An nnU-Net trained on CTPelvic1K, mostly unbroken pelvises, drew the bones; two annotators each corrected the draft and cut the fragments by hand, and a senior orthopaedic expert chose the better. Graded semi-automatic: the draft's prior may bias it, the authors write. Fragments under 500 mm³ were omitted, so a small loose piece of bone may be background.
Agreement. A third annotator redid the 30 hidden test CTs from the same draft: the three bones agreed at an IoU of 0.9996 (HD95 1.07 mm), about a millimetre at the surface, the surface Dice's tolerance.
Sides. hip_left lies on the patient's left in every case built. TotalSegmentator 2.18.0 on test case 006 scores a Dice of 0.93 for each hip bone and 0.81 for the sacrum, 0.90 once its vertebrae_S1, which the dataset's sacrum holds, is counted in.
Overlap. An image comparison with CTPelvic1K's CLINIC component found 31 shared images under other filenames, 4 of them cropped and flipped; seven built cases are CLINIC images (test 011, 041, 064, 065, 094; development 052, 067). CTPelvic1K_Clinic leaves those and other close candidates out by bone shape and finds no duplicate voxel-wise; the two still share a source cohort and are not independent populations.

		Accuracy			Cost
#	Method	Dice	Surface Dice	HD95 mm	Time s	RAM GB	GPU GB	Images
11–5	KonfAI segmentation app · TotalSegmentator (total), the authors' packageGPUapp	0.926 ±0.010	0.873 ±0.031	2.57 ±1.68		20.2	3.9
21–5	KonfAI segmentation app · TotalSegmentator (total), KonfAI appGPUapp	0.925 ±0.009	0.871 ±0.031	2.55 ±1.62	20	3.4	6.4
31–5	CT-FM whole-body segmentationGPUapp	0.907 ±0.037	0.837 ±0.066	21.88 ±15.05	101	12.6	6.4
41–5	KonfAI segmentation app · MRSegmentator, KonfAI appGPUapp3 failed	0.632 ±0.312	0.600 ±0.297	4.36 ±0.74		40.3	5.8
51–5	MONAI wholeBody CTGPUapp6 failed	0.367 ±0.339	0.332 ±0.307	2.60 ±0.70	19	4.1	6.8

The order averages each method's ranks on Dice, Surface Dice, HD95 mm, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 10 cases. Two neighbours are tied when a paired Wilcoxon test on the shared cases does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Workstation.

1 result not ranked: an older run, cases still missing, or a failure on every case

		Accuracy			Cost
#	Method	Dice	Surface Dice	HD95 mm	Time s	RAM GB	GPU GB	Machine	Images
	MONAI wholeBody CTGPUappolder run	0.912 ±0.010	0.835 ±0.035	3.05 ±0.69	12	2.7	10.1	Laptop

Dice: how much the organ contours overlap, 1 is perfect. Surface Dice: the share of the contours' surfaces within a tolerance of each other, 1 is perfect. HD95: distance between the contours' surfaces.

Case by case

Measure

Dice	1	2	3	4	5	6	7	8	9	10
KonfAI segmentation app · TotalSegmentator (total), the authors' package	0.922	0.929	0.888	0.931	0.924	0.934	0.930	0.935	0.932	0.931
KonfAI segmentation app · TotalSegmentator (total), KonfAI app	0.918	0.929	0.892	0.932	0.924	0.932	0.930	0.931	0.932	0.934
CT-FM whole-body segmentation	0.938	0.954	0.902	0.956	0.867	0.811	0.865	0.869	0.955	0.952
KonfAI segmentation app · MRSegmentator, KonfAI app	failed	0.904	0.892	0.900	failed	0.904	0.910	failed	0.906	0.907
MONAI wholeBody CT	0.912	failed	failed	0.927	0.912	failed	failed	0.921	failed	failed

Click a score to look at that result: the image, the truth in green and the method's map in red. Underlined scores have their image; for the others you see the case without the result.

About the data

The first test caseClick to look at every case.

Test cases: 10 test CTs drawn with seed 2026 among the 90 of the public training set outside development, and 10 development CTs. One CT a patient; the dataset names no centre or fracture type, so the draw has no strata. The challenge's 20 validation and 30 test CTs are not public.
Truth: contours by a program, verified by people the sacrum (1), the left hip bone (2) and the right hip bone (3), each the union of the dataset's fragments of that bone (prepare.py): a network's draft of the three bones corrected by hand and cut into fragments by hand, by two annotators independently, the better of the two chosen by a senior orthopaedic expert
Source: zenodo.org/records/10927452
pengwin.grand-challenge.org
arxiv.org/abs/2504.02382
github.com/YzzLiu/PENGWIN-example-algorithm/blob/main/evaluation_CT_core.py
Licence: CC BY 4.0

10 cases for tuning, never tested

017, 033, 055, 086, 075, 089, 052, 009, 067, 099