AI Is Learning How to Perfect Lighting in Photos

Table of Links

• Abstract and 1. Introduction
• 2. Related Work
• 3. SwitchLight and 3.1. Preliminaries
• 3.2. Problem Formulation
• 3.3. Architecture
• 3.4. Objectives
• 4. Multi-Masked Autoencoder Pre-training
• 5. Data
• 6. Experiments
• 7. Conclusion

Appendix

• A. Implementation Details
• B. User Study Interface
• C. Video Demonstration
• D. Additional Qualitative Results & References

3.3. Architecture

Illum Net. The network infers the lighting conditions in the given image captured in an HDRI format. Specifically, it computes the convolved HDRIs:

The network employs a cross-attention mechanism at its core, where predefined Phong reflectance lobes serve as queries, and the original image acts as both keys and values. Within this setup, the convolved HDRI maps are synthesized by integrating image information into the Phong reflectance lobe representation. Specifically, our model utilizes bottleneck features from the Normal Net as a compact image representation. Our approach simplifies the complex task of HDRI reconstruction by instead focusing on estimating interactions with known surface reflective properties.

We have empirically validated that it significantly enhances albedo prediction across a range of real-world scenarios.

Specular Net. The network infers surface attributes associated with the Cook-Torrance specular elements, specifically, the roughness α and Fresnel reflectivity f0. It uses a source image, predicted normal, and albedo maps as inputs.