This article presents a novel generative framework for learning shared latent representations across multimodal data, addressing key limitations of prior multimodal VAEs and introducing ShaLa - Shared Latent space modeling - as a principled approach to unify diverse modalities in a coherent latent space.
Motivation: from single-flow to multi-flow multimodal generation
Deep generative models have demonstrated remarkable success in generating high-fidelity outputs across a variety of single modalities. These advances have then rapidly extended to multimodal generation tasks, such as text-to-image, image-to-text, and other cross-modal flows. However, many of these approaches are tailored to specific generation directions (i.e., single-flow model) and often require dedicated models for each modality pair. To address this limitation, recent work has explored unified frameworks capable of handling multiple generation flows within a single model (i.e., multi-flow model).
Figure 1: Comparison of multimodal modeling paradigms. Left: Single-flow and multi-flow models typically learn direct mappings between modalities such as images, text, or views. To align with the way human cognition abstracts high-level concepts across different sensories, Multimodal Variational Autoencoders (VAEs) have emerged as a promising class of generative frameworks. However, a major bottleneck in multimodal VAEs lies in the design of the joint inference model.
Most existing approaches are built upon two dominant paradigms: Product of Experts (PoE) and Mixture of Experts (MoE). While PoE models enforce agreement across modalities, they struggle under missing-modality scenarios. Alternatively, MoE models are more robust to missing inputs but lack the expressivity needed to accurately capture complex joint posteriors. Besides this trade-off, the variational learning paradigm itself introduces an additional challenge: the prior-hole problem, a distributional mismatch between the aggregated joint posterior and the assumed prior. This gap often leads to poor sample quality and severely limits the effectiveness of multimodal VAEs in downstream generation tasks.
Foundations: PoE, MoE, and the prior-hole problem
In multimodal VAEs, a central question is how to design an effective joint inference distribution qφ(z|X). Product-of-Experts combines unimodal posteriors into a single joint distribution, enabling efficient training but facing robustness issues with missing modalities. Mixture-of-Experts, by contrast, allows partial inputs but can produce over-smoothed posteriors and weaker modality alignment, limiting expressiveness. The prior-hole problem arises when the aggregated posterior occupies only a narrow region of latent space, leaving large regions of the prior unaligned with the data distribution, which degrades sample quality and generation performance.
To place multimodal VAEs in context, note the broader trajectory of multimodal generative modeling: text-to-image synthesis, audio-visual generation, and multi-sensory robotics, among others. While many models excel in specific cross-modal tasks, there is a growing emphasis on unified frameworks that support multiple generation flows within a single model, i.e., multi-flow models. ShaLa explicitly targets learning a shared latent space to unify modalities rather than relying solely on flow-specific architectures.
The ShaLa framework: architecture and second-stage diffusion prior
ShaLa - Shared Latent space modeling - introduces a two-component approach to learning a robust, coherent shared latent representation across modalities. First, an architectural inference model encodes each modality into deterministic features and fuses them into a joint representation, acting as an information bottleneck that conditions the shared latent variable and promotes semantic alignment across modalities. This fusion enables rich cross-modal interactions while avoiding the combinatorial complexity of combining unimodal posteriors.
On top of this, ShaLa employs a second-stage latent diffusion model that serves as a flexible prior over the shared latent space. The diffusion model is trained to approximate the aggregated joint posterior with modality-aware guidance, thereby overcoming the prior-hole issue and enabling robust, coherent generation. The diffusion prior uses a denoising diffusion probabilistic model (DDPM) in the shared latent space, with a conditioning strategy that enhances flexibility for multimodal generation and cross-modal synthesis.
During training, ShaLa adopts a random conditioning strategy to improve flexibility in multimodal generation and optimizes the marginal likelihood of latent trajectories under the learned generative model. This enables joint and cross-modal sampling, with a controllable guidance scale that allows high-quality generation under both complete and incomplete observations.
Joint inference and the information bottleneck
ShaLa parameterizes the joint posterior qφi(z|X) as a single conditional Gaussian distribution, where its parameters are inferred from the entire set of input modalities. Each modality xi is encoded into a deterministic representation hi via a modality-specific encoder composed of convolutional and down-sampling layers. The information bottleneck h serves as a compact, high-level abstraction that captures modality-specific semantics while enabling rich cross-modal interactions through conditioning on the fused representation.
Training objectives and optimization
The prior-hole problem is addressed by the second-stage diffusion prior, which acts as a flexible, learnable prior over the shared latent space. Training involves maximizing the marginal likelihood of latent trajectories under the diffusion-augmented model, effectively aligning the aggregated posterior with a tractable prior and improving sample quality for downstream multimodal synthesis.
Joint and cross-modal sampling are supported, allowing the model to flexibly control conditioning strength using a guidance scale. This yields high-quality generation for complete observation scenarios as well as scenarios with missing modalities, where the model can rely on the learned shared latent structure and diffusion prior to fill in gaps meaningfully.
Baselines, datasets, and evaluation
ShaLa is benchmarked against representative shared latent variable generative models, including MVAE, MVTCAE, mmJSD, MoPoE, MMVAE, MMVAE+, CMVAE, and MVEBM. Datasets used include PolyMNIST, MST (MNIST-SVHN-Text), and CUB (Caltech UCSD Birds). Coherence and unconditional/conditional cross-modal generation are key evaluation metrics, with semantic coherence assessed by cross-modal classifiers ensuring modalities generated from the same latent code correspond to the same semantic class.
Figure 5: Unconditional and conditional generation on CUB demonstrates ShaLa’s ability to render semantically coherent outputs across modalities from latent samples and controlled conditioning.
Key contributions and outcomes
- ShaLa proposes a direct, expressive joint posterior qφ(z|X) that omits reliance on separate unimodal posteriors, enabling richer cross-modal interactions.
- A two-stage diffusion prior over the shared latent space mitigates the prior-hole problem and enhances synthesis quality for multimodal generation.
- Deterministic modality encoders produce fused representations that serve as robust information bottlenecks, aligning high-level semantics across modalities.
- Extensive experiments demonstrate superior coherence and synthesis quality across multiple benchmarks compared to state-of-the-art multimodal VAEs.
Practical implications and applications
ShaLa’s unified shared latent space supports flexible cross-modal generation and robust joint synthesis, with applications ranging from text-to-image and image-to-text to broader multi-sensory data integration in robotics, multimedia retrieval, and cross-modal analysis. The approach emphasizes semantic alignment over modality-specific details, enabling more reliable downstream tasks such as cross-modal retrieval, conditional synthesis, and multi-view data fusion when some modalities are missing.
Tables and figures
Table 1 compares PoE and MoE paradigms in terms of expressivity and cross-modal inference support. Figure 2 illustrates the prior-hole problem and how the aggregated posterior may occupy a narrow region of latent space. Figure 3 contrasts PoE/MoE with ShaLa’s joint posterior and modeling approach. Table 2 reports coherence metrics for unconditional and conditional cross-modal generation across benchmarks.
Additional notes: mathematical background
The true multimodal data distribution is pdata(X), and the generative model pθ(X) aims to approximate it. The latent prior p0(z) is usually standard Gaussian. Training optimizes the ELBO or marginal likelihood, with gradients involving expectations over pdata(X) and pθ(z|X). The aggregated posterior qφ(z) integrates over X with respect to pdata(X), capturing the joint information across modalities. ShaLa’s diffusion prior over zT and the conditioning on a fused representation address the mismatch between qφ(z) and p0(z), improving sample quality and coherence in downstream generation tasks.
