LCTSC

Radiotherapy planning CTs of the thorax of two patients of one institution, with five organs at risk contoured on both (Lung CT Segmentation Challenge, AAPM 2017), paired by us.

12 test pairslunginter-patientfixed CT, moving CTaligned once by the benchmark

Look at the images

Read before comparing (6)

Truth. The contours drawn in the clinic for treatment planning, reviewed and edited where needed to the RTOG 1106 atlas so that the 60 patients agree. The challenge's inter-rater study found the oesophagus and the spinal cord the most variable in Dice, the lungs and the heart in surface distance. Between two patients the contours are no verified correspondence.
Lungs. They leave out the tumour in most scans, with no guarantee on its extent, and may leave out collapsed lung; the hilar airways and the vessels over about 5 mm are out. The challenge masked those regions out of its scores; the corpus has no such mask, so a lung's overlap counts them.
Cord and oesophagus. The cord runs from below the cricoid, or the skull base, to L2, and may go past either end; the oesophagus from below the cricoid to the stomach. Where each stops differs between the two patients, which bounds the overlap of those thin organs whatever the registration.
Pairs. Two patients of one institution, S1, S2 or S3 in the name, one scan a patient, so one clinic's practice. On the series fetched, institution 1's headers name the 50% exhale phase of a respiratory CT, institution 2's the average of a 4D CT's ten phases; institution 3's say nothing.
Alignment. Each scan keeps its native grid. The moving image's and labels' headers carry one translation from CT intensities and body masks, no contour used. With no registration the 12 test pairs score a median mean-organ Dice of 0.450 (0.251 to 0.594) and a median HD95 of 37.3 mm.
Outside numbers. None: the pairing is ours.

		Accuracy			Smoothness		Cost
#	Method	Dice	Change	HD95 mm	SDlogJ	Folds %	Time s	RAM GB	GPU GB	Images
11–2	ConvexAdam (MIND)GPUapp	0.729 ±0.048	+0.276	23.45 ±5.70	0.121	1.17	30	4.2	10.1
21–2	DemonsCPUapp	0.691 ±0.048	+0.239	27.67 ±7.66	0.111	1.17	32	5.3
	No registration	0.453 ±0.064		37.27 ±7.82	0.000	0.00

The order averages each method's ranks on Dice, HD95 mm, SDlogJ, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 12 pairs. Two neighbours are tied when a paired Wilcoxon test on the shared pairs does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Laptop.

Dice: how much the organ contours overlap, 1 is perfect. HD95: distance between the contours' surfaces. SDlogJ: how irregular the deformation is, 0 is perfectly smooth. Folds: share of the image where the deformation folds over itself.

Pair by pair

Measure

Dice	1	2	3	4	5	6	7	8	9	10	11	12
ConvexAdam (MIND)	0.811	0.791	0.792	0.676	0.721	0.839	0.726	0.562	0.655	0.710	0.738	0.728
Demons	0.799	0.702	0.795	0.720	0.713	0.670	0.688	0.677	0.562	0.670	0.558	0.744
No registration	0.512	0.439	0.592	0.521	0.413	0.437	0.483	0.408	0.251	0.460	0.325	0.594

Click a score to look at that result: the fixed image, and over it the moving image after registration, with the annotations. Underlined scores have their image; for the others you see the pair without the result. A green cell gained against no registration on that pair, a red one lost.

About the data

The first test caseClick to look at every case.

Test pairs: The provider's 24 test patients, from sites S1, S2 and S3, all stay test, paired within site into 12 pairs. 10 development pairs use 20 of its 36 training patients, 4, 3 and 3 pairs by site, chosen by SHA256(2026:LCTSC:<patient>) within partition and site. No person, CT or RTSTRUCT crosses partitions.
Truth: contours by experts the dataset's manual contours of five organs at risk: 1 heart, 2 oesophagus, 3 left lung, 4 right lung, 5 spinal cord
Source: www.cancerimagingarchive.net/collection/lctsc
doi.org/10.7937/K9/TCIA.2017.3R3FVZ08
doi.org/10.1002/mp.13141
Licence: CC BY 3.0

10 pairs for tuning, never tested

Train-S1-002_Train-S1-001, Train-S1-007_Train-S1-006, Train-S1-008_Train-S1-003, Train-S1-010_Train-S1-009, Train-S2-001_Train-S2-012, Train-S2-002_Train-S2-010, Train-S2-004_Train-S2-011, Train-S3-003_Train-S3-008, Train-S3-006_Train-S3-004, Train-S3-011_Train-S3-009