fidoptTartanAir
Two consecutive colour frames of a camera flying through a rendered world, indoor or outdoor: frames k (fixed) and k + 1 (moving) at the middle of the first Hard trajectory of a TartanAir environment.
10 test pairsscenesame patientfixed PHOTO, moving PHOTO2D, distances in pxdelivered aligned
Read before comparing (8)
- Worlds. Photo-realistic Unreal Engine environments, indoor and outdoor, some with rain, snow, fog or night: a factory, a hospital, an office, a Japanese alley, an underwater scene, towns, forests and a desert. Three pairs of environments show one world under another light or season, or with the same rooms: one of each is kept. The setting reads "same patient" for one scene seen twice.
- Pairs. In each environment the Hard level's first trajectory, frames k and k + 1 at its middle, a rule set before any image was looked at: one pair an environment, since the frames of one world are not independent. Hard trajectories turn and shake the camera, so a pair moves 1.6 to 41 px on average.
- Truth. The provider computes the flow from the rendered depth and the camera's motion and masks the points that are occluded, leave the frame or lie on a moving object. Exact up to the renderer's depth, made by no person: graded synthetic and ranked apart.
- Points. A 16 x 16 grid of pixels on frame k, kept where the mask is 0 and the flow lands inside frame k + 1, 213 to 256 a pair, each with its image p + flow(p). Positions count from 0 at the first pixel's centre.
- Images. The PNG frames as delivered, red, green and blue a pixel, 640 x 480. No mask. Distances are in pixels; the rTRE divides them by the fixed image's diagonal, 800 px.
- Alignment. The frames come together, a moment apart, and get no starting transform: the camera's motion and the parallax it makes are the task.
- Baseline. With no registration the 10 test pairs score a TRE of 18.4 px (1.6 to 41.2), rTRE 0.023, oldtown closest and seasidetown farthest. Rain drops and snow flakes in some worlds do not follow the flow.
- Outside numbers. None: TartanAir ranks SLAM and visual odometry on its own test trajectories, not flow on these pairs.
No method has a result on every pair of this dataset yet.
Pair by pair
Click a score to look at that result: the fixed image, and over it the moving image after registration, with the annotations. Underlined scores have their image; for the others you see the pair without the result. A green cell gained against no registration on that pair, a red one lost.
About the data
- Test pairs
- 10 test environments drawn with seed 2026 among the 15 independent environments of TartanAir v1's training data, the other five for development. One pair an environment, from its Hard level.
- Truth
- landmarks a known deformation the provider's flow from frame k to frame k + 1, computed from the rendered depth and the camera's motion, at a 16 x 16 grid of pixels of frame k, kept where the provider's mask says the point is seen in both frames and the flow lands inside frame k + 1
- Source
- theairlab.org/tartanair-dataset
github.com/castacks/tartanair_tools
github.com/castacks/tartanair_tools/blob/master/data_type.md
huggingface.co/datasets/theairlabcmu/tartanair
arxiv.org/abs/2003.14338 - Licence
- CC BY 4.0
5 pairs for tuning, never tested
ocean, endofworld, gascola, hospital, amusement