fidopt Interactive view
Skin lesions on dermoscopy, 2D segmentation

ISIC 2018, skin lesions outlined on dermoscopy

A dermoscopic colour photograph of a pigmented skin lesion with the outline of its one primary lesion, made by hand or by an algorithm and reviewed by dermatologists (ISIC 2018 challenge, Task 1).

586 test casesskinimage DERMOSCOPY2D, distances in px
Look at the images
Read before comparing (7)
  • Images. The challenge's JPEGs as stored, in colour, of many sizes, from several dermatoscopes and institutions. No file gives a pixel size, so every distance is in pixels. No field-of-view mask is given: a round view's black corners are part of the image.
  • Masks. One connected region per image, of any size: test case ISIC_0036276's is the whole photograph. Each was traced by hand, flood-filled or made by an algorithm, all reviewed by dermatologists, with no record of which. Some outlines are polygons of a few dozen points; on the largest photographs some step in stairs of 30 to 40 pixels (test case ISIC_0015169).
  • Agreement. On 100 images of the 2016 challenge three experts agree at a Jaccard index of 0.743, 0.754 and 0.861 between pairs, a Dice of about 0.85 to 0.93: a method in that range is as close to the truth as another expert.
  • Tolerance. 10 pixels, about 0.3 % of a 3072-pixel photograph: at 1 pixel, under a polygon outline's precision, every method would score near 0, and on stair-stepped outlines even 10 pixels is finer than the truth. On the first ten test images an Otsu threshold scores a Dice of 0.81 but a surface Dice of 0.06.
  • Diagnoses. The draw keeps malignant lesions (the archive's diagnosis_1) and the others in proportion: the 586 test images hold 126 malignant and 460 other lesions.
  • Overlap. Test and development share no patient_id or lesion_id. Images without ids were compared on 64 x 64 grey thumbnails, flipped and rotated: no pair holding one correlates above 0.9. No other dataset of the corpus holds ISIC images.
  • Public models. The test masks were hidden during the 2018 challenge and are public since: methods trained on the 2018 test set, or on other ISIC archive releases holding these images, may have seen the test cases.

No method has a result on every case of this dataset yet.

Under a licence that is not open

Methods whose code or weights are released for non-commercial use, or under share-alike or custom terms: shown for reference, ranked nowhere.

AccuracyCost
#MethodDiceSurface DiceHD95 pxTime sRAM GBGPU GBMachineImages
VM-UNetCPUappApache-2.0 (code); weights: no licence statedolder run0.868 ±0.0110.162 ±0.016243.29 ±23.5161.9Laptop

Dice: how much the organ contours overlap, 1 is perfect. Surface Dice: the share of the contours' surfaces within a tolerance of each other, 1 is perfect. HD95: distance between the contours' surfaces.

Case by case

Dice123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586

Click a score to look at that result: the image, the truth in green and the method's map in red. Underlined scores have their image; for the others you see the case without the result.

About the data

The first test case of ISIC 2018, skin lesions outlined on dermoscopy
The first test caseClick to look at every case.
Test cases
The 586 of the challenge's 1000 test images that the ISIC archive gives under CC0, one image a patient or lesion by the archive's ids: 115 are left out for that, 299 for their CC-BY-NC licence. 30 development images come from the 2594 training images in the draw's order, seed 2026, without the 68 that share a patient or lesion with a test image or repeats within development.
Truth
contours by a program, verified by people the challenge's mask of the primary lesion: made by manual polygon tracing by a human expert, by a semi-automated flood fill with parameters an expert chose, or by a fully automated algorithm whose result a human expert reviewed and accepted, all reviewed and curated by practising dermatologists with expertise in dermoscopy (the challenge's Task 1 page, 'Ground Truth Provenance'); which method made which mask is not given
Licence
CC0 1.0 (the challenge's Task 1 licence, the masks' ATTRIBUTION.txt and LICENSE.txt, and the ISIC archive's licence of each image used)
30 cases for tuning, never tested

ISIC_0000141, ISIC_0012683, ISIC_0012378, ISIC_0000346, ISIC_0014454, ISIC_0013269, ISIC_0009915, ISIC_0015170, ISIC_0011336, ISIC_0000294, ISIC_0000522, ISIC_0000494, ISIC_0016028, ISIC_0010856, ISIC_0010466, ISIC_0012679, ISIC_0000299, ISIC_0012335, ISIC_0013334, ISIC_0000212, ISIC_0014349, ISIC_0013685, ISIC_0012395, ISIC_0015176, ISIC_0012710, ISIC_0014632, ISIC_0014558, ISIC_0013836, ISIC_0014150, ISIC_0012661