fidoptCOVID-19 CT Lung and Infection
One chest CT of a patient with COVID-19 pneumonia, with the left lung, the right lung and the infection drawn by hand and checked by a senior radiologist (COVID-19-CT-Seg, Ma et al. 2021).
10 test caseslungimage CTresearch only
Read before comparing (9)
- Source. The 20 CT volumes of the COVID-19 image data collection (Cohen et al.) as of 30 March 2020: ten from coronacases.org, patients of Wenzhou Medical University, and ten from Radiopaedia, eight patients from Iran, Italy, Ukraine, Turkey and Belgium.
- Values. The Coronacases scans are in Hounsfield units. The Radiopaedia scans are 8-bit, kept as delivered: the provider mapped the lung window [-1250, 250] HU to [0, 255], so nothing above 250 HU is told apart, and the window varies from scan to scan, so no exact conversion exists. A method that expects Hounsfield units reads the four Radiopaedia test scans on another scale.
- Geometry. The Radiopaedia scans all share 0.684 mm pixels and one in-plane origin, from a source the dataset's pages do not give; distances on them are in that geometry. Three of the four Radiopaedia test scans have 4 or 6 mm slices, where the HD95 and the surface Dice move by whole slices.
- Truth. Junior annotators drew the lungs and the infection by hand, slice by slice, two radiologists refined them and a senior chest radiologist verified them (Ma et al. 2021). The infection is 'all visibly affected regions of the lungs'. No agreement between readers is published.
- Labels. The infection is written over the lungs, so lung_left and lung_right are the lungs without their infected part; left is the patient's left, checked on all 18 cases. A whole-lung outline loses the infection's share: on radiopaedia_40_86625_0, where the infection fills 59% of the lungs, it scores about 0.58.
- Infection. It fills 0.01% to 59% of the lungs on the test scans, 4% at the median. On radiopaedia_29_86490_1 it is 0.5 cm3 in two pieces, so its Dice is all or nothing.
- Tolerance. The surface Dice uses 1 mm for the lungs and 3 mm for the infection, the provider's tolerances from the variation between two radiologists (truth.tolerance_mm).
- TotalSegmentator. Version 2.18, its lobes merged into lungs, scores a Dice of 0.966 and 0.958 for the left and right lung on coronacases_002, and 0.801 and 0.852 on radiopaedia_14_85914_0, in 8 bits. It makes no infection, and its lung HD95 of 20 to 29 mm comes from the infection's borders counting as lung surface.
- Outside numbers. The provider's nnU-Net, in five-fold cross-validation on these 20 scans with four training scans a fold, reaches a Dice of 0.65 for the left lung, 0.75 for the right and 0.61 for the infection on this map (Ma et al. 2021, table IV). The scans and labels have been public since April 2020: a model trained on the whole record has seen the test scans.
| Accuracy | Cost | |||||||
|---|---|---|---|---|---|---|---|---|
| # | Method | Dice | Surface Dice | HD95 mm | Time s | RAM GB | GPU GB | Images |
| 1 | KonfAI segmentation app · MRSegmentator, KonfAI appGPUapp2 of 3 structures | 0.904 ±0.096 | 0.773 ±0.124 | 15.68 ±6.10 | 10.8 | 8.3 | ||
| 22–7 | KonfAI segmentation app · TotalSegmentator (total), the authors' packageGPUapp2 of 3 structures | 0.872 ±0.101 | 0.696 ±0.157 | 21.39 ±11.58 | 18.6 | 3.9 | ||
| 32–7 | KonfAI segmentation app · TotalSegmentator (total), KonfAI appGPUapp2 of 3 structures | 0.859 ±0.109 | 0.687 ±0.171 | 20.04 ±10.29 | 25 | 3.1 | 5.7 | |
| 32–7 | lungmask · lobesGPUapp2 of 3 structures | 0.571 ±0.352 | 0.510 ±0.317 | 13.25 ±6.94 | 160 | 4.3 | 4.6 | |
| 52–7 | lungmaskGPUapp2 of 3 structures | 0.571 ±0.352 | 0.509 ±0.316 | 13.26 ±6.94 | 42 | 4.0 | 4.5 | |
| 62–7 | CT-FM whole-body segmentationGPUapp2 of 3 structures1 failed | 0.571 ±0.352 | 0.490 ±0.303 | 59.75 ±73.40 | 79 | 16.3 | 6.8 | |
| 72–7 | MONAI wholeBody CTGPUapp2 of 3 structures4 failed | 0.404 ±0.320 | 0.343 ±0.281 | 42.14 ±47.95 | 20 | 4.3 | 8.6 | |
The order averages each method's ranks on Dice, Surface Dice, HD95 mm, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 10 cases. Two neighbours are tied when a paired Wilcoxon test on the shared cases does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Workstation.
3 results not ranked: an older run, cases still missing, or a failure on every case
| Accuracy | Cost | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| # | Method | Dice | Surface Dice | HD95 mm | Time s | RAM GB | GPU GB | Machine | Images |
| lungmaskGPUapp2 of 3 structures4 failedolder run | 0.571 ±0.352 | 0.509 ±0.316 | 13.26 ±6.94 | 8 | 2.2 | 4.9 | Laptop | ||
| lungmask · lobesGPUapp2 of 3 structures4 failedolder run | 0.571 ±0.352 | 0.510 ±0.317 | 13.25 ±6.94 | 419 | 2.7 | 5.0 | Laptop | ||
| MONAI wholeBody CTGPUapp2 of 3 structuresolder run | 0.710 ±0.247 | 0.616 ±0.221 | 34.57 ±26.54 | 13 | 2.6 | 9.1 | Laptop | ||
Dice: how much the organ contours overlap, 1 is perfect. Surface Dice: the share of the contours' surfaces within a tolerance of each other, 1 is perfect. HD95: distance between the contours' surfaces.
Case by case
Click a score to look at that result: the image, the truth in green and the method's map in red. Underlined scores have their image; for the others you see the case without the result.
About the data
- Test cases
- 10 test scans drawn with seed 2026 among the first scans of the 18 patients, the two sources in proportion (6 Coronacases, 4 Radiopaedia), and the 8 others for development. A patient is one case: the second scans of two Radiopaedia patients, case 10's second series and case 29's second study, are left out.
- Truth
- contours by experts the two lungs and the infection, drawn by hand slice by slice, refined by two radiologists and checked by a senior one; the infection is written over the lungs, so lung_left and lung_right are the lungs without their infected part
- Source
- zenodo.org/records/3757476
doi.org/10.1002/mp.14676
arxiv.org/abs/2004.12537
github.com/JunMa11/COVID-19-CT-Seg-Benchmark
github.com/JunMa11/COVID-19-CT-Seg-Benchmark/blob/master/utils/COVID-19-Seg-Evaluation.py
academictorrents.com/details/136ffddd0959108becb2b3a86630bec049fcb0ff
github.com/ieee8023/covid-chestxray-dataset/blob/master/metadata.csv - Licence
- images CC BY-NC-SA 2.0 (their source); labels CC BY 4.0 on the Zenodo record, CC BY-NC-SA in its ReadMe and the paper
8 cases for tuning, never tested
radiopaedia_7_85703_0, coronacases_007, radiopaedia_27_86410_0, coronacases_006, radiopaedia_4_85506_1, coronacases_001, radiopaedia_10_85902_1, coronacases_003