fidoptPROMISE12
Prostate T2 MR of two patients of one centre, from the PROMISE12 challenge's test and live-challenge cases, paired by us, with the prostate segmented on both.
Read before comparing (11)
- Truth. An experienced reader drew each centre's references slice by slice, three readers in all; a second expert, who had read more than 1000 prostate MR scans, checked and corrected them.
- Readers. On the 30 test cases a less experienced blinded observer reached a Dice of 0.90 (0.86 at the base, 0.80 at the apex) and an HD95 of 5.6 mm: no registration reaches a Dice of 1, and an HD95 under 5 to 6 mm is within what two readers disagree on.
- Labels. The ranked truth is the whole gland, 22 to 73 mL in the test pairs (median 39 mL): it scores the gland's outline, not its zones or the rest of the pelvis.
- Organs. TotalSegmentator's organs, run by us on each image, are shown (dice_organs), never ranked. It finds 12 to 18 structures on the HK, UCL and RUNMC images but is often lost on BIDMC's: on live-Case14 its prostate misses the gland, on test-Case11 it finds a fifth of it. Small fragments at the field's edges pull dice_organs down.
- Alignment. HK, BIDMC and RUNMC come in the scanner's frame, slices tilted up to 17 degrees; UCL's files carry no orientation. Each pair gets one elastix translation on mutual information inside each body, no label used, written in the moving image's and labels' headers. The rotation, the difference in gland size (up to 2.6 times the volume) and the deformation are left to the methods.
- Baseline. With no registration the 10 test pairs score a prostate Dice of 0.62 (0.39 to 0.81), an HD95 of 10.8 mm (7.5 to 18.3) and an organ Dice of 0.37 (0.02 to 0.61).
- Scanners. HK (Siemens, 1.5 T) and BIDMC (GE, 3 T) used an endorectal coil, which fills the rectum behind the gland; UCL and RUNMC did not. A pair joins two patients of one centre. Slices lie 2.2 to 2.8 mm apart for BIDMC and 3.3 to 3.6 mm for the others.
- Values. 16-bit integers as delivered, with no common scale: at most 1150 to 1427 for UCL but 4321 to 31044 for BIDMC, where the coil makes the tissue next to it far brighter.
- Exposure. The 2023 record gives the reference of all 100 cases. The pairs come from the 50 test and live cases the challenge held out, so a model trained on its 50 training cases has seen none of them. A case enters one pair only; the record does not say whether two cases may come from one patient.
- Overlap. Seven Radboud images, test-Case00, 01, 03 and 04 and live-Case15, 17 and 19, are training cases of the decathlon's Task05 Prostate, voxel for voxel (prostate_20, 21, 24, 25, 32, 35, 37); MSD_Prostate leaves them out.
- Outside numbers. None: the pairing is ours.
| Accuracy | Smoothness | Cost | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| # | Method | Dice | Change | HD95 mm | SDlogJ | Folds % | Time s | RAM GB | GPU GB | Images |
| 1 | deedsCPUapp | 0.814 ±0.031 | +0.198 | 6.22 ±0.73 | 0.086 | 0.24 | 18 | 1.0 | ||
| 22–5 | greedyCPUapp | 0.695 ±0.062 | +0.079 | 10.15 ±2.78 | 0.070 | 0.00 | 24 | 1.2 | ||
| 32–5 | ConvexAdam (MIND)GPUapp | 0.657 ±0.066 | +0.041 | 10.02 ±2.42 | 0.099 | 0.31 | 12 | 2.3 | 1.7 | |
| 42–5 | NiftyRegCPUapp | 0.633 ±0.084 | +0.017 | 14.50 ±2.83 | 0.368 | 6.79 | 21 | 0.6 | ||
| 52–5 | DemonsCPUapp | 0.627 ±0.099 | +0.011 | 13.11 ±3.09 | 0.542 | 8.44 | 5 | 0.9 | ||
| No registration | 0.616 ±0.090 | 10.84 ±2.31 | 0.000 | 0.00 | ||||||
The order averages each method's ranks on Dice, HD95 mm, SDlogJ, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 10 pairs. Two neighbours are tied when a paired Wilcoxon test on the shared pairs does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Laptop.
Dice: how much the organ contours overlap, 1 is perfect. HD95: distance between the contours' surfaces. SDlogJ: how irregular the deformation is, 0 is perfectly smooth. Folds: share of the image where the deformation folds over itself.
Pair by pair
Click a score to look at that result: the fixed image, and over it the moving image after registration, with the annotations. Underlined scores have their image; for the others you see the pair without the result. A green cell gained against no registration on that pair, a red one lost.
About the data
- Test pairs
- 10 test pairs drawn with seed 2026 among the 24 pairs of the challenge's test and live cases, stratified by centre, and 10 development pairs, the last of the same seeded order (3 UCL, 3 RUNMC, 2 HK, 2 BIDMC). UCL and HK each leave one case out of the pairing (live-Case00, test-Case27).
- Truth
- contours by experts organs by a program the challenge's reference segmentation of the prostate on each image, one label
- Source
- zenodo.org/records/8026660
promise12.grand-challenge.org/Details
doi.org/10.1016/j.media.2013.12.002 - Licence
- none stated beyond the record's LICENSE.TXT: 'You can use this data for your research', citing the paper being mandatory (Zenodo: Other (Attribution))
10 pairs for tuning, never tested
test-Case14_test-Case20, test-Case01_test-Case00, test-Case29_live-Case05, test-Case12_test-Case08, live-Case04_test-Case15, live-Case17_test-Case03, test-Case22_test-Case28, live-Case12_test-Case10, live-Case03_test-Case18, live-Case15_test-Case05