LCTSC, thoracic organs at risk

Radiotherapy planning CT of one patient's thorax, with five organs at risk contoured by hand (AAPM Lung CT Segmentation Challenge 2017).

24 test caseslungimage CT

Look at the images

Read before comparing (2)

Cases. Both CTs and their contours of each LCTSC test pair, shared with that dataset rather than stored twice. A moving CT and its labels keep the intensity-derived start written in their headers.
Check. On test CT Test-S1-101, TotalSegmentator matches the expert contours at a Dice of 0.757 for the heart, 0.751 for the oesophagus, 0.977 for each lung and 0.842 for the spinal cord: a sanity check on one case, not a baseline.

		Accuracy			Cost
#	Method	Dice	Surface Dice	HD95 mm	Time s	RAM GB	GPU GB	Machine	Images
1	lungmaskGPUapp2 of 5 structures	0.944 ±0.022	0.800 ±0.059	7.48 ±3.24	14	3.5	4.4	Workstation
2	lungmask · lobesGPUapp2 of 5 structures	0.931 ±0.044	0.793 ±0.066	6.82 ±3.03	63	3.8	4.5	Workstation
33–4	MONAI wholeBody CTGPUapp3 of 5 structures	0.858 ±0.023	0.713 ±0.057	13.33 ±7.75	16	3.1	16.8	Laptop
43–4	KonfAI segmentation app · MRSegmentator, KonfAI appGPUapp4 of 5 structures	0.857 ±0.013	0.640 ±0.042	12.45 ±1.46		11.6	8.3	Workstation
55–6	KonfAI segmentation app · TotalSegmentator (total), the authors' packageGPUapp	0.835 ±0.016	0.657 ±0.044	19.81 ±4.13		19.1	6.3	Workstation
65–6	KonfAI segmentation app · TotalSegmentator (total), KonfAI appGPUapp	0.835 ±0.015	0.654 ±0.043	33.16 ±10.55	40	8.2	5.7	Workstation
7	KonfAI segmentation app · IMPACTSeg body, KonfAI appGPUapp1 of 5 structures	0.731 ±0.043	0.660 ±0.052	49.05 ±17.82		1.7	3.5	Workstation

The order averages each method's ranks on Dice, Surface Dice, HD95 mm, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 24 cases. Two neighbours are tied when a paired Wilcoxon test on the shared cases does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it.

4 results not ranked: an older run, cases still missing, or a failure on every case

		Accuracy			Cost
#	Method	Dice	Surface Dice	HD95 mm	Time s	RAM GB	GPU GB	Machine	Images
	KonfAI segmentation app · TotalSegmentator (total), KonfAI appGPUappolder run	0.835 ±0.015	0.654 ±0.043	33.16 ±10.55	30	3.8	12.5	Laptop
	lungmaskGPUapp2 of 5 structuresolder run	0.944 ±0.022	0.800 ±0.059	7.48 ±3.24	54	3.6		Laptop
	lungmask · lobesGPUapp2 of 5 structuresolder run	0.931 ±0.044	0.793 ±0.066	6.82 ±3.03	85	2.1	5.0	Laptop
	MONAI wholeBody CTGPUapp3 of 5 structuresfailed on every case	0.000 ±0.000	0.000 ±0.000					Workstation

Dice: how much the organ contours overlap, 1 is perfect. Surface Dice: the share of the contours' surfaces within a tolerance of each other, 1 is perfect. HD95: distance between the contours' surfaces.

Case by case

Measure

Dice	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
lungmask	0.391	0.390	0.390	0.391	0.384	0.388	0.395	0.393	0.376	0.378	0.378	0.378	0.360	0.381	0.355	0.389	0.334	0.390	0.393	0.356	0.378	0.313	0.388	0.395
lungmask · lobes	0.391	0.390	0.390	0.391	0.384	0.388	0.395	0.393	0.376	0.378	0.378	0.378	0.360	0.381	0.355	0.389	0.334	0.390	0.393	0.356	0.378	0.193	0.388	0.395
MONAI wholeBody CT	0.540	0.551	0.537	0.540	0.534	0.543	0.502	0.548	0.509	0.544	0.530	0.531	0.520	0.509	0.494	0.415	0.490	0.523	0.504	0.497	0.513	0.462	0.473	0.542
KonfAI segmentation app · MRSegmentator, KonfAI app	0.703	0.710	0.713	0.705	0.698	0.705	0.661	0.711	0.682	0.693	0.694	0.706	0.693	0.672	0.657	0.695	0.685	0.701	0.687	0.665	0.665	0.607	0.644	0.703
KonfAI segmentation app · TotalSegmentator (total), the authors' package	0.861	0.832	0.874	0.854	0.850	0.868	0.813	0.865	0.855	0.863	0.846	0.866	0.864	0.841	0.816	0.867	0.788	0.823	0.817	0.779	0.787	0.728	0.806	0.875
KonfAI segmentation app · TotalSegmentator (total), KonfAI app	0.861	0.835	0.870	0.852	0.855	0.860	0.818	0.865	0.855	0.863	0.843	0.866	0.864	0.837	0.815	0.863	0.789	0.823	0.817	0.781	0.786	0.740	0.810	0.876
KonfAI segmentation app · IMPACTSeg body, KonfAI app	0.155	0.120	0.160	0.156	0.148	0.169	0.143	0.150	0.163	0.161	0.159	0.164	0.168	0.157	0.158	0.163	0.113	0.115	0.134	0.113	0.112	0.112	0.155	0.160

Click a score to look at that result: the image, the truth in green and the method's map in red. Underlined scores have their image; for the others you see the case without the result.

About the data

The first test caseClick to look at every case.

Test cases: All 24 released Test patients, each once, from LCTSC's test pairs. Ten development Train patients, 4/3/3 across S1/S2/S3, chosen from LCTSC's development patients in the PsCBCT family's SHA256 order; the other ten stay family training exclusions but are not cases here.
Truth: contours by experts the dataset's manual contours of five organs at risk: 1 heart, 2 oesophagus, 3 left lung, 4 right lung, 5 spinal cord
Source: www.cancerimagingarchive.net/collection/lctsc
doi.org/10.7937/K9/TCIA.2017.3R3FVZ08
doi.org/10.1002/mp.13141
Licence: CC BY 3.0

10 cases for tuning, never tested

Train-S1-007_Train-S1-006/moving, Train-S1-008_Train-S1-003/fixed, Train-S1-008_Train-S1-003/moving, Train-S1-010_Train-S1-009/moving, Train-S2-001_Train-S2-012/fixed, Train-S2-002_Train-S2-010/moving, Train-S2-004_Train-S2-011/moving, Train-S3-006_Train-S3-004/fixed, Train-S3-006_Train-S3-004/moving, Train-S3-011_Train-S3-009/moving