Figure 1. RGB-NIR Camera and the NIR visible dyes. Top Left: The inside structure of the camera. Right: Sample images captured by the RGB CCD sensor and NIR CCD sensor respectively. Top Middle: The relative transmittance of our RGB CCD sensor and NIR CCD sensor (yellow). Bottom Middle: The absorbance of the NIR visible dyes respect to various wavelength.

I present the first ground truth data set of nonrigidly deforming real-world scenes (both long and short video sequences). To construct ground truth for the RGB sequences, the proposed system simultaneously capture Near-Infrared (NIR) image sequences where the dense markers - visible only in NIR - represent the ground truth positions, allowing comparison between the RGB tracked positions and the formation of error metrics. This novel ground truth construction protocol may also be adopted to capture other types of deformable objects, thus opening ground truth opportunities in other difficult-to-track problems. Unlike previous datasets containing nonrigidly deforming sequences using synthetic data, the capture of real-world objects yields realistic photometric effects - such as blur and illumination change - as well as occlusion and complex deformations. A public evaluation website is constructed to allow for ranking of RGB image based optical flow and other dense tracking algorithms, with varying statistical measures. Furthermore, I present the first RGB-NIR multispectral optical flow formulation allowing for overall optimisation of the optical flow energy by maximizing the distinguishing information from both the RGB and the complementary NIR channels. In the experiments I evaluate eight existing optical flow methods on this new dataset, as well as examine the proposed multispectral optical flow algorithm by varying the input channels across RGB, NIR and RGB-NIR.

Our Evaluation Website

W. Li, D. Cosker, Z. Lv and M. Brown, Nonrigid Optical Flow Ground Truth for Real-World Scenes with Time-Varying Shading Effects, IEEE Robotics and Automation Letters (RA-L’16), 2016. [PDF]

W. Li, D. Cosker, Z. Lv and M. Brown, Dense Nonrigid Ground Truth for Optical Flow in Real-World Scenes, IEEE Conference on Automation Science and Engineering, 2016. [PDF]