fidoptFIVES, retinal vessels drawn by hand
A colour photograph of the retina, 2D, with its vessels painted by two trained annotators and corrected by an ophthalmic practitioner: one of the 800 images of FIVES (Second Affiliated Hospital of Zhejiang University), 200 each normal, AMD, diabetic retinopathy and glaucoma.
Read before comparing (10)
- Images. 800 photographs of 573 patients aged 4 to 83, taken 2016 to 2021 at the Ophthalmology Centre of the Second Affiliated Hospital of Zhejiang University with a Topcon TRC-NW8, 50 degrees, centred on the macula: 2048 x 2048 PNGs, in colour as stored. No pixel size is given, so distances are in pixels. No field-of-view mask is given or made.
- Truth. Keeping only the pixels both annotators painted, and the rule not to join a vessel that looks broken, lean the masks towards fewer vessel pixels: many thin vessels seen on the photographs are not drawn (around the disc of test case test_101_G), and a method that finds them is scored down for them. On the test images vessels cover 7.1% of the pixels on average, from 5.8% on glaucoma to 8.4% on normal eyes.
- Agreement. By the paper, two junior annotators agree at a Dice of 0.924 (0.879 to 0.982) before review, a senior and a junior at 0.961 and one annotator with themself a week or more apart at 0.968 (both on 40 images).
- Tolerance. 3 pixels, the same share of the width as STARE's one pixel of 700 (0.15%). The truth grown or thinned by one pixel keeps a Dice of 0.94 and 0.93 and a surface Dice of 1.0 at 3 pixels, and 0.99 at one.
- Start. On the 200 test images Little W-Net with its authors' DRIVE model, which has seen no FIVES image, scores a Dice of 0.72 (median 0.81) and a surface Dice of 0.69 at 3 pixels; it finds no vessel on ten dark glaucoma images. The dark pixels of the green channel (Otsu) score a Dice of 0.18.
- Strata. Each patient has one clear diagnosis; diabetic retinopathy includes laser scars. The provider's split takes 150 of each diagnosis for training and 50 for testing, chosen by hand on image quality, yet the test split holds more images scored poor on illumination, blur or contrast: 67 of 200 (17 of the 50 normal) against 141 of 600.
- Repeats. The paper says each eye gives one photograph, yet the test split shows some eyes twice. test_32_A and test_48_A are one file, its vessels drawn twice (a Dice of 0.993 between the two masks); test_22_A and test_23_A are near copies sharing one mask. test_3_A and test_33_A, test_9_A and test_35_A, test_14_A and test_15_A, and test_84_D and test_85_D are one eye photographed twice. test_48_A and test_23_A, the copies, are left out of the test cases (draw.held_out); the other pairs stay, two photographs of one eye.
- Patients. The deposit names no patient, but most photographs carry a line of Chinese text cut at the left edge, which reads like the end of a name. 21 consecutive pairs of normal test images share that text and show a right and a left eye. Four training images whose text matches a test image of the same diagnosis, on the other eye, are kept out of development (draw.held_out). 33 of the 230 images carry no text; a match of the vessel masks that finds the six pairs above finds no development image of a test eye.
- Outside numbers. The test masks are public since 2022 and FIVES is widely used to train vessel networks: a public model trained on all 800 images has seen the test cases.
- Overlap. No other dataset of the corpus holds images of this hospital: STARE (San Diego), FIRE (Thessaloniki), FIMD (Tianjin) and COph100 (Ostrava) come from elsewhere.
| Accuracy | Cost | |||||||
|---|---|---|---|---|---|---|---|---|
| # | Method | Dice | Surface Dice | HD95 px | Time s | RAM GB | GPU GB | Images |
| 1 | Little W-NetCPUapp | 0.716 ±0.032 | 0.684 ±0.034 | 187.61 ±43.53 | 3 | 0.6 | ||
The order averages each method's ranks on Dice, Surface Dice, HD95 px, so the first one is not always the best in every column. ± is the 95% interval of the mean over the 198 cases. Two neighbours are tied when a paired Wilcoxon test on the shared cases does not find the upper one better on Dice (5%): the places a method could hold are then shown under its own. The best value of each column is in colour. Click a row to see its images, a column title to sort by it. Time and memory measured on Laptop.
Dice: how much the organ contours overlap, 1 is perfect. Surface Dice: the share of the contours' surfaces within a tolerance of each other, 1 is perfect. HD95: distance between the contours' surfaces.
Case by case
Click a score to look at that result: the image, the truth in green and the method's map in red. Underlined scores have their image; for the others you see the case without the result.
About the data
- Test cases
- The provider's test split, 50 images of each diagnosis, less two copies: 198 test images. 30 development images drawn with seed 2026 among the 600 of its training split, less four that may be a test patient's other eye (notes). One image a case.
- Truth
- contours by experts the provider's vessel masks: each image painted pixel by pixel by two of 24 trained medical staff, the pixels both marked kept, then reviewed and corrected by one of three ophthalmic practitioners (Jin et al. 2022, 'Vessel annotation')
- Source
- figshare.com/articles/figure/FIVES_A_Fundus_Image_Dataset_for_AI-based_Vessel_Segmentation/19688169
doi.org/10.6084/m9.figshare.19688169.v1
doi.org/10.1038/s41597-022-01564-3
pmc.ncbi.nlm.nih.gov/articles/PMC9352679 - Licence
- CC BY 4.0 (the figshare deposit's licence)
30 cases for tuning, never tested
train_90_A, train_471_N, train_417_G, train_256_D, train_138_A, train_116_A, train_130_A, train_229_D, train_198_D, train_370_G, train_100_A, train_205_D, train_294_D, train_580_N, train_314_G, train_406_G, train_581_N, train_573_N, train_129_A, train_218_D, train_590_N, train_301_G, train_166_D, train_192_D, train_154_D, train_158_D, train_578_N, train_91_A, train_233_D, train_62_A