fidopt Interactive view
MR to CT, synthesis

Multimodal-HC, whole-body MR to CT

Whole-body Dixon in-phase MR (source) and low-dose CT (target) of one healthy adult, from the vertex to mid-femur, on two scanners in two rooms, the MR registered onto the CT by the corpus (Multimodal-HC, Rigshospitalet).

10 test caseswhole bodysource MR, target CT
Look at the images
Read before comparing (12)
  • Images. The low-dose CT reconstructed for the PET's attenuation correction (Siemens Biograph Vision Quadra, 120 kVp, 512 x 512 voxels of 1.52 mm, slices of 2 mm, free breathing, arms down) and the whole-body Dixon in-phase MR (Siemens MAGNETOM Vida 3 T, four breath-hold stations composed into one volume of 1.3 x 1.3 x 3 mm voxels) of one healthy volunteer, taken the same day for 96 of the 100 volunteers (up to four cases may be weeks apart; the release does not say which).
  • Pool. The release's train split holds 80 subjects. MultimodalHC_MRCT's 20 are left out, as their CT is that set's fixed image, which every method sees, and MultimodalHC_PETCT_Syn's 20, as their CT is that set's target; so are sub-084 and sub-087, whose registrations failed, and sub-018, which has no MR_IMPACT in Valentin Boussot's registration, so that its head cannot be redone (below). The other 37 make the pool.
  • Registration. The CT is the reference and is never resampled: it is cut to the box the registration covered, voxels untouched, and the MR is resampled into it once, linearly, through a stored transform. The corpus registered the MR onto the CT with IMPACT-Reg 1.8.7's preset ConvexAdam_Composite (presets VBoussot/ImpactReg at 09d2a191), regularization weight 5, its other settings the preset's: an affine stage on mutual information, then coarse and fine stages on MIND, seed 42. Both images were masked by their bodies (IMPACTSeg, VBoussot/ImpactSeg at 6fd8149), and the start was a translation matching the two bodies' area slab by slab along the body axis.
  • Head. A smooth field does not turn the head on the neck, and the head lies otherwise in the MR's head coil than on the CT's table. Above the neck the field is refined onto Valentin Boussot's own IMPACT registration of the release (huggingface.co/datasets/VBoussot/Multimodal-HC at f504977, made for TotalSynth): an elastix 5.3.1 B-spline (normalized correlation, bending energy, a 16 mm grid, four levels) of the corpus's registered MR onto his, kept above the CT's C3 and dropped below its C7, a cosine ramp between (C3 and C7 from the provider's TotalSegmentator map of the CT). His registration has no MR_IMPACT for sub-018, so sub-018 is held out and the seeded order's next subject, sub-002, takes its place.
  • Transforms. transforms/<case>.npz holds the box, the start, the affine part of the field and the rest of it, the head's refinement included, on a 4 mm grid (16-bit integers, tenths of a mm, stored as differences along its three axes), 47 MB for the 20 cases (the refined head compresses less well than the smooth field). Below the neck the stored field is within 0.34 to 0.63 mm of the registration's at its 99th percentile, the tenths adding at most 0.05 mm a component, and prepare.py applies it: building again gives the same files without registering.
  • Check. With the provider's TotalSegmentator maps (total on the CT, total_mr on the MR, the 46 to 48 structures MultimodalHC_MRCT keeps), carried as prepare.py carries the MR, the mean Dice on the test subjects goes from 0.18 to 0.39 at the start to 0.59 to 0.73, the brain's to 0.88 to 0.93 (0.56 to 0.92 before the head was refined). TotalSegmentator 2.18's vertebrae_mr on the registered MR, against the CT's map, gives C1 0.37 to 0.54 and C2 0.47 to 0.74 (C1 0.00 to 0.48 before, on the nine other test subjects). Before the head was refined, the registration's own check (TotalSegmentator 2.18 on both images, 67 to 73 structures) put the bones' centroids 3.4 to 5.9 mm apart on average on the test subjects.
  • Failures. A registration failed when its mean Dice fell more than 0.02 below the start's, or when a bone, the brain or the spinal cord at 0.5 or more at the start lost more than 0.25: sub-084 (vertebrae T11 and T12) and sub-087 (T7 to T9 and T12), where the field follows the diaphragm and drags the lower thoracic spine. They are held out.
  • Residual. The head is closer, not exact (C1 at 0.37 to 0.54). The MR is held at deep inspiration where the CT breathes freely, so the lung bases and the diaphragm differ, and the field drags T10 to L1 a little (sub-014, sub-059). The bladder fills and the bowel moves. CT soft tissue facing air in the registered MR is 0.6 to 3.3% of the mask on the test cases (2.8 to 7.5% before the arms left the mask and the head was refined), where a method faithful to the MR loses 5 to 33 HU of MAE. Behind the head, outside the CT's body and the mask, the refinement can pull the MR's scalp into the air (sub-002, sub-025, sub-042, sub-093). What the registration leaves counts as error for every method.
  • Folding. At most 0.14% of the body has a Jacobian determinant of 0 or less (on the 4 mm grid), 0.04% on the test subjects.
  • Mask. The provider's TotalSegmentator body of the CT (trunk and extremities, the table left out) without the arms, inside the box, where the transform lands in the MR's grid: 42 to 74 litres on the test cases. The arms leave the CT's or the MR's field of view and lie otherwise on the two tables; they are taken out as in TotalSynth's Multimodal-HC pairs (prepare.py's arms(), from Valentin Boussot's arms.py). The faces were blanked by the provider, differently on the two images (MultimodalHC_MRCT's notes): the CT's body map leaves its own blanked face out, the MR's blanked region can fall inside the mask.
  • Values. The CT keeps the whole Hounsfield units its header's scaling was made from (int16, as MultimodalHC_MRCT writes it). The MR is the scanner's in-phase image in arbitrary units, no bias-field correction, the steps between its four stations visible.
  • Baseline. Inside the mask on the ten test cases, a constant at the CT's median there (-69 to 19 HU) scores an MAE of 102 to 151 (128 on average) HU, a PSNR of 23.1 to 26.9 (24.7) dB and an SSIM of 0.28 to 0.42 (0.34), by the corpus's evaluation. On the ten development cases, ImpactSynth_MR (TotalSynth's MR-to-CT model, a KonfAI app) scores an MAE of 51 to 104 HU (74 on average) where the constant scores 121 to 173 (141): a method faithful to the MR beats the constant on every case. The ten test cases take 1.2 GB as built.

No method has a result on every case of this dataset yet.

Under a licence that is not open

Methods whose code or weights are released for non-commercial use, or under share-alike or custom terms: shown for reference, ranked nowhere.

AccuracyCost
#MethodMAEPSNR dBSSIMAnatomy DiceAnatomy HD95 mmTime sRAM GBGPU GBMachineImages
KonfAI synthesis app · TotalSynth MR to CT, KonfAI appGPUapptrained on CC-BY-NC-4.0 dataolder run75.1 ±7.528.06 ±0.950.745 ±0.0190.656 ±0.03014.39 ±1.10503.94.5Workstation

MAE: mean absolute error of a synthetic image against its target, inside the body, in the image's units. PSNR: the same by signal-to-noise ratio, higher is better. SSIM: the same by structural similarity, 1 is perfect. Anatomy Dice and HD95: TotalSegmentator run on the synthetic and on the target image, the structures compared; shown, not ranked.

Case by case

MAE12345678910

Click a score to look at that result: the target image, and over it the method's synthetic image. Underlined scores have their image; for the others you see the case without the result.

About the data

Test cases
10 test subjects drawn with seed 2026 by sex (5 women, 5 men) among the 37 of the open release's 80 left once MultimodalHC_MRCT's and MultimodalHC_PETCT_Syn's subjects and three registrations failed or not refined are out (draw.held_out), then 10 development subjects (5 and 5). No subject is in another dataset of the corpus or in both splits.
Truth
target image by a program the CT for attenuation correction in Hounsfield units, cut to the box the registration covered; the MR registered onto it by the corpus with IMPACT-Reg 1.8.7 (ConvexAdam_Composite, regularization weight 5), refined above the neck onto Valentin Boussot's IMPACT registration, the transform kept in transforms/
Licence
CC BY 4.0
10 cases for tuning, never tested

sub-073, sub-000, sub-071, sub-058, sub-050, sub-078, sub-086, sub-088, sub-082, sub-057