Summary of full paper, presented at IWAR'99, San Francisco, October 20-21, 1999
The full article is published in the printed "Proceedings of the 2nd IWAR'99" and copyright protected by IEEE Computer Society.

Photometric Image-Based Rendering for Virtual Lighting Image Synthesis

Yasuhiro MUKAIGAWA, Sadahiko MIHASHI and Takeshi SHAKUNAGA


A concept named Photometric Image-Based Rendering (PIBR) is introduced for a seamless augmented reality. The PIBR is defined as Image-Based Rendering which covers appearance changes caused by the lighting condition changes, while Geometric Image-Based Rendering (GIBR) is defined as Image-Based Rendering which covers appearance changes caused by the view point changes. The PIBR can be applied to image synthesis to keep photometric consistency between virtual objects and real scenes in an arbitrary lighting condition. We analyze the conventional IBR algorithms, and formalize the PIBR in the whole IBR framework. A specific algorithm is also presented for realizing the PIBR. The photometric linearization makes a controllable framework for the PIBR, which consists of four processes; (1) separation of environmental illumination effects, (2) estimation of lighting directions, (3) separation of specular reflections and cast-shadows, and (4) linearization of self-shadows. After the photometric linearization of input images, we can synthesize any realistic images which include not only diffuse reflections but also self-shadows, cast-shadows and specular reflections. Experimental results show that realistic images can be successfully synthesized with keeping photometric consistency.

Copyright (c) 1999 IEEE. Reprinted, with permission, from IWAR'99 proceedings.

This web page contains only experimental results. If you want to know the details of our PIBR method, please read the proceedings of IWAR'99.

Photometric Image-Based Rendering

First, we show image syntheses of a glossy ceramic pot. Keeping a halogen light in the long distance from the pot, we took 27 images with changing the lighting source position. The input images are linearized, and the principal component analysis was accomplished to make three optimal base images. Several lighting directions corresponding to the coefficients were specified, and virtual images were synthesized. We show one of the synthesized images as follows. The left is a synthesized image without linear factors. Both diffuse reflections and self-shadows are correctly synthesized. The right is a synthesized image with linear factors, which were selected by the nearest neighbor method. We can see that the realistic images with the appropriate surface properties can be synthesized by the proposed PIBR method.

Synthesized image without nonlinear factors Synthesized image with nonlinear factors
Synthesized image by the PIBR.
LEFT: Synthesized image without nonlinear factors.
RIGHT: Synthesized image with nonlinear factors.
You can see a movie sequence by clicking on each image. (MPEG: 2MB)

Application for Augmented Reality

Next, we show some results of mixing virtual objects and real scenes with keeping the photometric consistency. The left images are real scenes in which the virtual object is to be mixed. The virtual object is synthesized by the PIBR to have the same lighting direction as the real scene. The right images are the mixed images, a mouse is a virtual object while a cup and a book are real ones. Since the photometric property is consistent between the real scene and the virtual object, the mixed image looks realistic.

Real scene Mixed image
Real scene Mixed image
Real scene Mixed image
Real scene Mixed image
Real scene Mixed image
Real scene Mixed image

This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by sending a blank email message to By choosing to view this document, you agree to all provisions of the copyright laws protecting it.