This New Tech Makes Faces in Photos Look Incredibly Realistic

Table of Links

• Abstract and 1. Introduction
• 2. Related Work
• 3. SwitchLight and 3.1. Preliminaries
• 3.2. Problem Formulation
• 3.3. Architecture
• 3.4. Objectives
• 4. Multi-Masked Autoencoder Pre-training
• 5. Data
• 6. Experiments
• 7. Conclusion

Appendix

• A. Implementation Details
• B. User Study Interface
• C. Video Demonstration
• D. Additional Qualitative Results & References

3. SwitchLight

We introduce SwitchLight, a state-of-the-art framework for human portrait relighting, with its architectural overview presented in Fig. 2. We first provide foundational concepts in Sec. 3.1, and define the problem in Sec. 3.2. This is followed by detailing the architecture in Sec. 3.3, and lastly, we describe the loss functions used in Sec. 3.4.

3.1. Preliminaries

In this section, vectors n, v, l, and h are denoted as unit vectors. Specifically, n represents the surface normal, v is the view direction, l is the incident light direction, and h is the half-vector computed from l and v. The dot product is clamped between [0..1], indicated by ⟨·⟩.

Image Rendering. The primary goal of image rendering is to create a visual representation that accurately simulates the interactions between light and surfaces. These complex interactions are encapsulated by the rendering equation:

A surface intrinsically exhibits both diffuse and specular reflections. The diffuse component uniformly scatters light, ensuring consistent illumination regardless of the viewing angle. In contrast, the specular component is viewing angle-dependent, producing shiny highlights that are crucial for achieving a photorealistic effect.

Lambertian Diffuse Reflectance. Lambertian reflectance, a standard model for diffuse reflection, describes a uniform light scatter irrespective of the viewing angle. This ensures a consistent appearance in brightness:

Here, σ is the albedo, indicating the intrinsic color and brightness of the surface.

Cook-Torrance Specular Reflectance. The Cook-Torrance model, based on microfacet theory, represents surfaces as a myriad of tiny, mirror-like facets. It incorporates a roughness parameter α, which allows precise rendering of surface specular reflectance:

In this model, D is the microfacet distribution function, describing the orientation of the microfacets relative to the half-vector h, G is the geometric attenuation factor, accounting for the shadowing and masking of microfacets, and F is the Fresnel term, calculating the reflectance variation depending on the viewing angle, where f0 is the surface Fresnel reflectivity at normal incidence. A lower α value implies a smoother surface with sharper specular highlights, whereas a higher α value indicates a rougher surface, resulting in more diffused reflections. By adjusting α, the Cook-Torrance model can depict a range of specular reflections.

Image Formation. Upon the base rendering equation, we include the diffuse and specular components of the BRDF and derive a unified formula:

where E(l) denotes the incident environmental lighting. This formula represents the core principle that an image is a product of interplay between the BRDF and lighting. To further clarify this concept, we introduce a rendering function R, succinctly modeling the process of image formation:

It is important to note that since the BRDF is a function of surface properties, as detailed in Eqn. 3 and 4, we can now clearly understand that image formation is essentially governed by the interaction of surface attributes and lighting.