Brain MR, inter-subject, automatic labels

LUMIR (Learn2Reg 2024), training images

Brain T1 MR of two healthy subjects from OpenBHB, skull-stripped and linearly registered to MNI152 by the LUMIR organisers.

30 test pairsbraininter-patientfixed MR, moving MRdelivered alignedresearch only

Look at the images

Read before comparing (4)

Truth. LUMIR keeps its own labels and landmarks private. The labels here are made by FastSurfer v2.5.4, a segmentation network no method of the benchmark uses, so the dataset is ranked apart with the others whose truth comes from a network.
Baseline. With no registration the 30 test pairs score a Dice of 0.46 (0.39 to 0.51) and an HD95 of 4.95 mm.
Overlap. OpenBHB includes CoRR, which holds the NKI scans Mindboggle-101 also draws from. After an affine alignment none of the 23 NKI brains of Mindboggle-101 in this corpus correlates with any of the 40 LUMIR brains of the 20 pairs built first more than two different brains do (0.48 at most, against up to 0.50), where its two people scanned twice reach 0.957 and 0.979.
Outside numbers. None: with its labels and landmarks private, LUMIR's challenge numbers cannot be reproduced here.

		Accuracy			Smoothness		Cost
#	Method	Dice	Change	HD95 mm	SDlogJ	Folds %	Time s	RAM GB	GPU GB	Images
1	ConvexAdam (MIND)GPUapp	0.620 ±0.007	+0.159	3.64 ±0.09	0.049	0.01	9	2.4	2.0
2	DemonsCPUapp	0.594 ±0.010	+0.133	3.96 ±0.15	0.026	0.00	6	1.0
	No registration	0.461 ±0.012		4.95 ±0.18	0.000	0.00

The order averages each method's ranks on Dice, HD95 mm, SDlogJ, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 30 pairs. Two neighbours are tied when a paired Wilcoxon test on the shared pairs does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Laptop.

Dice: how much the organ contours overlap, 1 is perfect. HD95: distance between the contours' surfaces. SDlogJ: how irregular the deformation is, 0 is perfectly smooth. Folds: share of the image where the deformation folds over itself.

Pair by pair

Measure

Dice	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30
ConvexAdam (MIND)	0.598	0.639	0.631	0.641	0.639	0.651	0.615	0.583	0.616	0.627	0.613	0.621	0.651	0.613	0.597	0.591	0.606	0.612	0.571	0.620	0.650	0.624	0.609	0.625	0.639	0.612	0.628	0.618	0.621	0.628
Demons	0.558	0.621	0.600	0.627	0.615	0.642	0.603	0.567	0.603	0.587	0.586	0.597	0.649	0.586	0.542	0.562	0.598	0.598	0.547	0.589	0.629	0.605	0.567	0.595	0.567	0.575	0.617	0.597	0.581	0.610
No registration	0.389	0.476	0.476	0.505	0.491	0.507	0.505	0.485	0.491	0.443	0.413	0.479	0.492	0.450	0.441	0.418	0.459	0.420	0.432	0.441	0.495	0.475	0.433	0.497	0.420	0.441	0.478	0.461	0.446	0.472

Click a score to look at that result: the fixed image, and over it the moving image after registration, with the annotations. Underlined scores have their image; for the others you see the pair without the result. A green cell gained against no registration on that pair, a red one lost.

About the data

Test pairs: 30 test pairs drawn with seed 2026 among the 1,692 pairs of LUMIR's training images, then 30 development pairs. LUMIR's validation pairs are not used: they hold OASIS and AFIDs subjects that L2R_OASIS and AFIDs already hold.
Truth: contours by a program FastSurfer's segmentation run by us on each image: the subcortical structures and the DKT cortical parcellation
Source: github.com/JHU-MedImage-Reg/LUMIR_L2R
doi.org/10.1016/j.neuroimage.2022.119637
Licence: CC BY-NC-SA 3.0 (OpenBHB, the most restrictive terms of its cohorts); LUMIR's own files say CC BY 4.0

30 pairs for tuning, never tested

1004_1506, 2624_0403, 2203_0518, 2230_2486, 2283_0487, 3097_2159, 1809_0533, 3326_2166, 1507_1573, 1910_0188, 2724_2464, 3132_0011, 0463_1109, 0736_0713, 1282_0395, 0306_0101, 1546_2101, 0123_2193, 2979_1407, 0550_2847, 1973_1989, 1763_2250, 2182_3342, 1024_0927, 0594_0802, 2427_1120, 1139_1738, 2813_2162, 0899_1484, 3106_2649