RibSeg v2

Chest-abdomen CT of two patients from one hospital, most with rib fractures (RibFrac), their 24 ribs labelled one by one (RibSeg v2), paired by us.

30 test pairsthorax, abdomeninter-patientfixed CT, moving CTaligned once by the benchmarkresearch only

Look at the images

Read before comparing (8)

Truth. RibSeg's morphological pipeline made the labels of 519 scans, from a threshold at 200 HU; on the 127 where it failed, a senior radiologist placed ten points along each rib and the rib was grown from that curve. People refined every step, and two junior and one senior radiologist confirmed every map in consensus: graded semi-automatic.
Ribs. The labels stop where the bone stops: costal cartilages, sternum, clavicles, scapulae and spine are not labelled. A rib is thin, so a few millimetres of error cost much of its overlap: Dice is harsher here than on an organ, and HD95 tells more at the start.
Missing ribs. The scorer counts only the ribs both patients hold: 238 of the 240 rib pairs of the test cases, the 12th pair lying beyond RibFrac475's partial scan. The description sheet lists RibFrac404 as missing ribs 7 to 12, while its labels hold all 24 at their usual volumes.
Fractures. Of the test patients whose fracture labels are public, 16 hold 2 to 25 fractures each and RibFrac475 none; those of RibFrac539, 581 and 587, in RibFrac's test set, are not public. Displaced or segmental fractures and metal (RibFrac283) leave rib pieces that no registration between two patients matches exactly.
Alignment. Two patients' scanner frames lie up to 40 cm apart on the test pairs. The moving header carries one elastix translation on mutual information inside each body; no label is used and no voxel resampled. ANTs' rigid, tried on 8 test pairs, left the matching ribs further apart (centroids 31 mm apart against 21 mm).
Baseline. With no registration the 30 test pairs leave a rib Dice of 0.04 (0.00 to 0.18) and an HD95 of 36.8 mm (20.2 to 73.8): the rib cages lie together, yet the ribs barely overlap.
Images. RibFrac's CTs from Huadong Hospital (Fudan University, Shanghai), on a GE and a Siemens scanner; the files do not say which, so the draw has no strata. Some scans read -3024 outside the reconstructed circle, left as they are.
Outside numbers. None: the pairing is ours, and the pairs mix the three parts of RibFrac's own split, which is for fracture detection.

		Accuracy			Smoothness		Cost
#	Method	Dice	Change	HD95 mm	SDlogJ	Folds %	Time s	RAM GB	GPU GB	Images
11–2	DemonsCPUapp	0.142 ±0.034	+0.099	33.68 ±4.93	0.088	0.07	81	12.6
21–2	NiftyRegCPUapp	0.312 ±0.068	+0.269	36.24 ±5.47	0.774	1.55	458	8.7
	No registration	0.043 ±0.014		36.80 ±4.70	0.000	0.00

The order averages each method's ranks on Dice, HD95 mm, SDlogJ, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 30 pairs. Two neighbours are tied when a paired Wilcoxon test on the shared pairs does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Laptop.

1 result not ranked: an older run, pairs still missing, or a failure on every pair

		Accuracy			Smoothness		Cost
#	Method	Dice	Change	HD95 mm	SDlogJ	Folds %	Time s	RAM GB	GPU GB	Machine	Images
	ConvexAdam (MIND)GPUapp15/30 pairs	0.301 ±0.078	+0.258	29.23 ±5.43	0.133	1.43	52	7.3	21.5	Laptop

Dice: how much the organ contours overlap, 1 is perfect. HD95: distance between the contours' surfaces. SDlogJ: how irregular the deformation is, 0 is perfectly smooth. Folds: share of the image where the deformation folds over itself.

Pair by pair

Measure

Dice	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30
Demons	0.054	0.084	0.250	0.029	0.000	0.187	0.007	0.189	0.048	0.257	0.198	0.138	0.253	0.129	0.225	0.119	0.103	0.144	0.054	0.362	0.129	0.178	0.204	0.109	0.213	0.023	0.067	0.244	0.057	0.201
NiftyReg	0.271	0.293	0.546	0.051	0.000	0.419	0.047	0.626	0.110	0.481	0.529	0.307	0.557	0.257	0.392	0.301	0.158	0.324	0.154	0.653	0.287	0.373	0.414	0.208	0.356	0.047	0.128	0.472	0.161	0.431
ConvexAdam (MIND)	0.234							0.570		0.370				0.303		0.272	0.253	0.300	0.139	0.534		0.327	0.371		0.363	0.079	0.072			0.322
No registration	0.011	0.022	0.053	0.022	0.000	0.029	0.002	0.059	0.018	0.105	0.024	0.034	0.068	0.036	0.106	0.042	0.033	0.027	0.017	0.183	0.031	0.048	0.040	0.040	0.051	0.007	0.024	0.088	0.016	0.047

Click a score to look at that result: the fixed image, and over it the moving image after registration, with the annotations. Underlined scores have their image; for the others you see the pair without the result. A green cell gained against no registration on that pair, a red one lost.

About the data

Test pairs: 30 test pairs drawn with seed 2026 among 327, then 29 development pairs. prepare.py orders the 654 patients with rib labels by sha256("2026:RibSeg:<patient>") and pairs them two by two, each patient in one pair. The six the paper calls unqualified are left out: RibFrac452, 485 and 490 have no rib labels, RibFrac462, 471 and 487 labels the description sheet marks wrong.
Truth: contours by a program, verified by people the 24 ribs one by one, RibSeg v2's labels: 1 to 12 the left ribs from the first to the twelfth, 13 to 24 the right ones
Source: github.com/HINTLab/RibSeg
doi.org/10.1109/TMI.2023.3313627
arxiv.org/abs/2210.09309
docs.google.com/spreadsheets/d/1lz9liWPy8yHybKCdO3BCA9K76QH8a54XduiZS_9fK70
zenodo.org/records/3893508
zenodo.org/records/3893498
zenodo.org/records/3893496
zenodo.org/records/3993380
ribfrac.grand-challenge.org
arxiv.org/abs/2402.09372
doi.org/10.1016/j.ebiom.2020.103106
Licence: CC BY-NC 4.0 (the CTs); the labels carry no licence of their own

29 pairs for tuning, never tested

14_638, 137_449, 570_199, 26_585, 628_66, 649_302, 114_582, 128_609, 298_363, 555_154, 537_291, 616_90, 106_476, 430_270, 20_601, 253_278, 51_533, 72_348, 98_189, 460_575, 442_489, 650_426, 134_211, 469_31, 610_80, 73_172, 63_68, 75_381, 514_339