Submit a method

Three ways to put a method on the leaderboards. All are scored by the same code, on the same cases.

Route 1 · Results

Send what you computed

Run your method on the test cases at home and hand over one result per case: a displacement field, a label map or a synthetic image. Accuracy only: time and memory cannot be compared from another machine.

Route 2 · KonfAI app

Package your method as an app

A folder that runs your code at a fixed version: a wrapper, its settings and a short description. The benchmark runs it: accuracy, time and memory, and anyone can run it again with one command.

Route 3 · Container

Send an image we run

For a method whose code is not public. Run by the maintainers on a reference machine, without network and without the annotations. Planned, not open yet.

A registration's displacement field

One file per test pair: a float32 image on the fixed image's grid with one component per dimension (3, or 2 on a 2D dataset), the displacement u of each fixed voxel in mm (or the dataset's own unit, pixels for photographs without a pixel size) along the world (ITK, LPS) axes, so that the fixed point x corresponds to x + u(x) in the moving image. Learn2Reg displacements, in voxels, are read too.

fidopt score my_method Lung250M --fields /path/to/fields         # <pair>.nii.gz, in mm
fidopt score my_method L2R_OASIS --fields /path/to/disp --voxels # Learn2Reg displacements

The KonfAI app

MyMethod/model.py           # the wrapper: calls your code, returns its result in the benchmark's convention
MyMethod/Prediction.yml     # its settings
MyMethod/app.json           # what it is and its licence; a segmentation names its labels
MyMethod/requirements.txt   # your packages, pinned

konfai-apps infer MyMethod -i fixed.nii.gz -i moving.nii.gz -o out   # registration: DisplacementField
konfai-apps infer MyMethod -i image.nii.gz -o out                    # segmentation: Segmentation, synthesis: Synthetic

The method is given the images and the masks the dataset allows, never the annotations. A registration writes its displacement field, from a pair already roughly aligned: run the deformable part. A segmentation writes its label map, its labels named in app.json and matched with the dataset's by name. A synthesis writes the image of the target modality on the input's grid.

What a result must have

A method is ranked on a dataset when it has a result on every test case. A missing case counts against it.
Settings tuned for a task are stated, with the cases they were tuned on. A method tuned on a dataset's test cases is not ranked there.
The version of the method is fixed and named, so that the result can be reproduced.