Poster, presented at IWAR'99, San Francisco, October 20-21, 1999

Interactive Virtual Relighting and Remodeling of Real Scenes

Celine Loscos, Marie-Claude Frasson , George Drettakis, Bruce Walter


Lighting design is often tedious due to the required physical manipulation of real light sources and objects. As an alternative, we present an interactive system [Loscos99] to virtually modify the lighting and geometry of scenes with both real and synthetic objects, including mixed real/virtual lighting and shadows.

  1. Reflectance Estimation
  2. In our method, real scene geometry is first approximately reconstructed from photographs. Additional images are taken from a single viewpoint with a real light in different positions to estimate reflectance. A filtering process is used to compensate for inaccuracies, and per image reflectances are averaged to generate an approximate diffuse reflectance image for the given viewpoint, removing shadows in the process.

    An example of extracted reflectance is given in Figure 1 (b) corresponding to the photograph of Figure 1(a). Other approaches could be used [Yu99] for diffuse and specular reflectance extraction. This reflectance estimate is used to initialise a global illumination hierarchical radiosity system, representing real-world secondary illumination. Direct illumination from lights is calculated separately using ray-casting and a table for efficient reuse of data where appropriate. Figure 1 (c) shows shadow reprojection using the estimated reflectance (b).


    Figure 1. (a) Original captured image, (b) reflectance estimate by merging images of many light positions, (c) reprojected image with no scene/lighting modification, (d) the relit image after door removal in 2.9 s. (resolution 512x340). (e) Virtual chair added in 3.4 s., and (f) a virtual light added in 6.6 s. (On a 250Mhz R10000 SGI).

  3. Interactive Modifications
  4. After completing these preprocessing steps, we are ready to interactively model and relight our scene. When modifying the lighting or the geometry of the scene (either real, virtual or both), direct and indirect light are efficiently updated. The regions of the image for which direct illumination must be recomputed are efficiently identified and shaft data structures are used for dynamic global illumination updates. These same structures also allow efficient recomputation of indirect light.

    Our system permits interactive modification of light emission and object positions, all with mixed real/virtual illumination effects. Real objects can be virtually removed using texture-filling algorithms for reflectance estimation.

    Examples of the results are shown in Fig. 1 (d)-(f): the removal of a real door, the addition of a virtual object and a virtual light. All changes take a few seconds.

    An accompanying paper was presented at the Eurographics workshop on Rendering '99 at Granada, Spain. The paper and some web movies of interactive use of the system can be found at: .


    Thanks to Xavier Granier and Pierre Poulin. iMAGIS is joint project of CNRS, INRIA, the Universite Joseph Fourier and the Institut National Polytechnique de Grenoble.


    [Loscos99] C. Loscos, M.-C. Frasson, G. Drettakis, B. Walter, X. Granier, and P. Poulin. Interactive virtual relighting and remodeling of real scenes. In Rendering Techniques '99 (10th Eurographics Workshop on Rendering). Springer-Verlag, June 1999.

    [Yu99] Y. Yu, P. Debevec, J. Malik, and T. Hawkins. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In Proceedings of SIGGRAPH '99 , 1999.

    This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by sending a blank email message to By choosing to view this document, you agree to all provisions of the copyright laws protecting it.