Researchers at Cornell Tech in New York published a study called “Crowdsampling the Plenoptic Function”, which was presented during the European Conference on Computer Vision, held virtually from 23 to 28 August, where they demonstrated 4D images of some of the sights most famous in the world. 4D images have 3D depth and show visual variations according to the variation of time.
To be able to give the 4D effect to the images, the researchers use the approach of the plenotic function, which is the action of describing the appearance of something from all possible points of view in space and time. This would be achieved more comfortably if they put hundreds of cameras fixed around a scene, recording the image variations day and night. However, this solution would not be practical (or cheap).
For this reason, the researchers were inspired by an animation technique created by Walt Disney Studios in the 1930s, which superimposed layers of transparencies to create a 3D effect without redrawing all the objects in a scene. The difference is that, now, the technique was applied to real photos, instead of drawings.
This new scene representation to interpolate the appearance in four dimensions was named “Deep Multiplane Images”.
Using publicly available images
To create the 4D images, researchers would need to overlay thousands of images of the same scene to create just one with four dimensions. For that, they used the images of tourist spots publicly available on sites like Flickr and Instagram.
Through deep learning techniques, it was possible to catalog tens of thousands of photographs taken by tourists, to create final images that can be viewed from different points of view, and at different times of the day, as if we were seeing the monument in person.
The study was partially funded by philanthropist Eric Schmidt, a former Google CEO, and his wife.