Controllable Portrait Relighting using Neural Rendering
Main Article Content
Abstract
We propose a novel controllable portrait relighting method leveraging a light vector conditioning framework within a lightweight neural rendering network. Our approach enables explicit control over lighting direction and intensity while preserving facial identity and realistic details. We validate the method on a toy dataset of synthetic portraits under diverse lighting conditions, demonstrating im- proved relighting quality and user controllability.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/
References
JungWoo Chae, Hyunin Cho, Sooyeon Go, Kyungmook Choi, and Youngjung Uh. Semantic image syn- thesis with unconditional generator. Advances in Neural Information Processing Systems, 36:32999– 33011, 2023.
Sumit Chaturvedi, Mengwei Ren, Yannick Hold-Geoffroy, Jingyuan Liu, Julie Dorsey, and Zhixin Shu. Synthlight: Portrait relighting with diffusion model by learning to re-render synthetic faces. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pages 369–379, 2025.
Yajing Chen, Fanzi Wu, Zeyu Wang, Yibing Song, Yonggen Ling, and Linchao Bao. Self-supervised learning of detailed 3d face reconstruction. IEEE Transactions on Image Processing, 29:8696–8705, 2020.
Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 145–156, 2000.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In European conference on computer vision, pages 694–711. Springer, 2016.
Ira Kemelmacher-Shlizerman, Supasorn Suwajanakorn, and Steven M Seitz. Illumination-aware age progression. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3334–3341, 2014.
Hua Ma and Junichi Yamaoka. Sensequins: Smart textile using 3d printed conductive sequins. In Pro- ceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–13, 2022.
Wojciech Matusik, Chris Buehler, Ramesh Raskar, Steven J Gortler, and Leonard McMillan. Image- based visual hulls. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 369–374, 2000.
Rohit Pandey, Sergio Orts-Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhe- mann, Paul E Debevec, and Sean Ryan Fanello. Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph., 40(4):43–1, 2021.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intel- ligence, volume 32, 2018.
Pratul P Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng. Learning to syn- thesize a 4d rgbd light field from a single image. In Proceedings of the IEEE international conference on computer vision, pages 2243–2251, 2017.
Yu-Ying Yeh, Koki Nagano, Sameh Khamis, Jan Kautz, Ming-Yu Liu, and Ting-Chun Wang. Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation. ACM Transactions on Graphics (TOG), 41(6):1–21, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effec- tiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18643–18652, 2022.
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region- adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5104–5113, 2020.