Realistically inserting digital characters into real scenes shot with a camera, Iron Man, the tiger in Life of Pi, the CGI versions of Neo and Agent Smith in The Matrix Reloaded, has been a core visual effects challenge since the early days of computer-generated imagery.  Today, this is often done using Image-Based Lighting: illuminating the CGI elements with panoramic high dynamic range images shot on the set or location.  First presented at SIGGRAPH conference in the late 1990’s, the technique was just recently recognized with an Academy Award for Pioneering Technical Achievement.

Image-Based Lighting has also been the genesis of today’s virtual production LED Volumes, seen in movies such as Gravity and TV Series such as The Mandalorian to light actors with the light of the virtual environments they are meant to be seen within.  Improved techniques use multispectral LED lights which include broad-spectrum white and amber LEDs to improve how scene elements are lit on set, and time-multiplexed lighting to record live action in multiple lighting conditions simultaneously for relighting in postproduction.

The latest LED volumes, including the Light Dome at Eyeline Studios, are being used not just for real-world image-based lighting, but to acquire training data for machine learning models to relight scenes in postproduction.  These new techniques are making it possible to estimate lighting and relight actors all in software, and combined with advances in volumetric capture, to change camera and lighting in postproduction.