Distribution Matching Vae

Overview

DMVAE generalizes beyond conventional Gaussian priors by explicitly aligning the encoder's latent distribution with an arbitrary reference distribution. The framework enables matching with distributions derived from self-supervised learning, diffusion noise, or other priors—moving beyond rigid architectural constraints.

When to Use

• Image generation models where latent distribution affects quality
• Scenarios exploring optimal latent structures for specific tasks
• Applications where self-supervised learning distributions outperform Gaussians
• Models needing flexibility in distribution choice
• Seeking efficient training (64 epochs on ImageNet)

When NOT to Use

• Models already achieving satisfactory results with Gaussian priors
• Fixed, rigid latent assumptions are acceptable
• Scenarios where distribution flexibility adds complexity without benefit

Core Technique

Explicit distribution matching for flexible latent space design:

# Distribution Matching VAE
class DistributionMatchingVAE:
    def __init__(self, encoder, decoder, reference_distribution=None):
        self.encoder = encoder
        self.decoder = decoder
        self.reference_dist = reference_distribution or self.default_gaussian()

    def forward(self, x):
        """
        Encode to latent space and reconstruct.
        Explicitly match latent distribution to reference.
        """
        # Encode
        latent, encoder_params = self.encoder(x)

        # Reconstruction
        recon = self.decoder(latent)

        return recon, latent, encoder_params

    def compute_distribution_matching_loss(self, latent_samples):
        """
        Explicitly align encoder latent distribution with reference.
        This replaces implicit KL divergence from fixed Gaussian prior.
        """
        # Measure distribution discrepancy
        # e.g., Wasserstein distance, Maximum Mean Discrepancy, etc.
        matching_loss = self.compute_distribution_distance(
            latent_samples,
            self.reference_dist
        )

        return matching_loss

    def train_step(self, batch):
        """
        DMVAE training combines reconstruction with distribution matching.
        """
        x = batch

        # Forward pass
        recon, latent, encoder_params = self.forward(x)

        # Reconstruction loss (pixel-space quality)
        recon_loss = torch.nn.functional.mse_loss(recon, x)

        # Distribution matching loss (latent structure)
        dist_matching_loss = self.compute_distribution_matching_loss(latent)

        # Combined objective
        total_loss = recon_loss + self.beta * dist_matching_loss

        return total_loss

    def use_ssl_derived_distribution(self, ssl_model, dataset):
        """
        Match encoder to self-supervised learning distribution.
        SSL distributions balance reconstruction fidelity and efficiency.
        """
        # Extract SSL features as reference distribution
        ssl_features = []
        for batch in dataset:
            features = ssl_model.encode(batch)
            ssl_features.append(features)

        ssl_features = torch.cat(ssl_features, dim=0)

        # Fit reference distribution to SSL features
        # (e.g., mixture of Gaussians, normalizing flow, etc.)
        self.reference_dist = self.fit_distribution_to_features(
            ssl_features
        )

        return self.reference_dist

    def use_diffusion_noise_distribution(self, diffusion_model):
        """
        Match encoder to diffusion model noise distribution.
        Enables alignment with denoising training paradigms.
        """
        # Diffusion models implicitly define noise distributions
        # at different timesteps
        noise_samples = diffusion_model.sample_noise_distribution()
        self.reference_dist = self.fit_distribution_to_features(
            noise_samples
        )

        return self.reference_dist

    def compute_distribution_distance(self, samples, reference):
        """
        Measure discrepancy between sample distribution and reference.
        Supports multiple distance metrics for flexibility.
        """
        if self.metric == 'wasserstein':
            # Wasserstein distance via optimal transport
            distance = self.wasserstein_distance(samples, reference)
        elif self.metric == 'mmd':
            # Maximum Mean Discrepancy
            distance = self.maximum_mean_discrepancy(samples, reference)
        elif self.metric == 'ks':
            # Kolmogorov-Smirnov test
            distance = self.ks_distance(samples, reference)
        else:
            raise ValueError(f"Unknown metric: {self.metric}")

        return distance

The framework systematically investigates which latent distributions are optimal for image synthesis. SSL-derived distributions provide superior performance for bridging high-fidelity synthesis with computational efficiency (gFID 3.2).

Key Results

• gFID 3.2 on ImageNet with 64 training epochs
• Self-supervised learning distributions outperform Gaussian priors
• Competitive with standard VAE approaches
• Flexible distribution choice enables optimization
• Reduced training time while maintaining quality

Implementation Notes

• Reference distribution can be arbitrary (SSL features, diffusion, etc.)
• Matching constraint replaces fixed KL divergence
• Multiple distance metrics supported (Wasserstein, MMD, KS)
• Training converges in significantly fewer epochs
• Can compare different distribution choices systematically

References

• Original paper: https://arxiv.org/abs/2512.07778
• Focus: Latent distribution design in VAEs
• Domain: Generative models, image synthesis