Mutual Information Constrained Variational Framework for Identifiable Representation Disentangling
Main Article Content
Abstract
Disentangled representation learning aims to uncover underlying generative factors in data such that each latent dimension corresponds to a distinct factor of variation. Ensuring identifiability of these factors remains a central challenge. We propose a novel variational framework incorporating mutual information constraints to encourage independence among latent dimensions, coupled with a theoretical guarantee of identifiability. We validate our approach on a synthetic toy dataset with known factors (size, color intensity, rotation) and demonstrate improved disentanglement metrics and qualitative interpretations.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/
References
Gulcin Baykal, Melih Kandemir, and Gozde Unal. Disentanglement with factor quantized variational autoencoders. arXiv preprint arXiv:2409.14851, 2024.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in backslash beta-vae. arXiv preprint arXiv:1804.03599, 2, 2018.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adver- sarial nets. Advances in neural information processing systems, 29, 2016.
Emilien Dupont. Learning disentangled joint continuous and discrete representations. Ad- vances in neural information processing systems, 31, 2018.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In 6th International Conference on Learning Representations, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
Aapo Hyv ̈arinen, Ilyes Khemakhem, and Hiroshi Morioka. Nonlinear independent compo- nent analysis for principled disentanglement in unsupervised deep learning. Patterns, 4(10), 2023.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial intelligence and statistics, pages 460–469. PMLR, 2017.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International conference on artificial intelligence and statistics, pages 2207–2217. PMLR, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Yexiong Lin, Yu Yao, Xiaolong Shi, Mingming Gong, Xu Shen, Dong Xu, and Tongliang Liu. Cs-isolate: Extracting hard confident examples by content and style isolation. Advances in Neural Information Processing Systems, 36:58556–58576, 2023.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Scho ̈lkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 4114–4124. PMLR, 2019.