Apple researchers propose FAE, a single-layer framework to adapt visual encoders for image generation

Apple researchers—Yuan Gao, Chen Chen, Tianrong Chen, and Jiatao Gu—present FAE (Feature Auto-Encoder), a framework designed to bridge a fundamental gap in generative modeling. Visual generative models like diffusion models typically operate in compressed latent spaces to optimize training efficiency and sample quality. In recent years, researchers have shown interest in leveraging pre-trained visual representations—such as those from self-supervised vision transformers—either by aligning them within VAEs (Variational Autoencoders) or directly within the generative model itself. However, this adaptation has proven difficult because of a core mismatch: representation encoders for understanding tasks benefit from high-dimensional latents that capture diverse hypotheses for masked or occluded regions, while generative models require low-dimensional latents that must faithfully preserve the noise injected during the generation process. Prior approaches have relied on complex objectives and architectural modifications to bridge this gap.

FAE simplifies the solution through an elegant design: it uses as little as a single attention layer to adapt pre-trained visual representations into generation-friendly low-dimensional latents while retaining enough information for both reconstruction and understanding. The core mechanism couples two separate deep decoders—one trained to reconstruct the original high-dimensional feature space, and a second that takes the reconstructed features as input for image generation. This decoupling allows each decoder to optimize for its specific task without forcing a single latent representation to serve both purposes. The framework is generic: it can be instantiated with various self-supervised encoders such as DINO and SigLIP, and it can be plugged into two distinct generative model families—diffusion models and normalizing flows.

On ImageNet 256×256 benchmarks, FAE demonstrates strong empirical results. With classifier-free guidance (CFG), FAE's diffusion model attains a near–state-of-the-art FID (Fréchet Inception Distance) of 1.29 at 800 epochs and 1.70 at 80 epochs. Without CFG, FAE reaches state-of-the-art FID of 1.48 at 800 epochs and 2.08 at 80 epochs. These numbers reflect both high sample quality and efficient learning, as the model achieves competitive performance even in the fast-learning regime (80 epochs). The paper also references complementary work in related areas: Representation Tokenizer (RepTok), which represents images using a single continuous latent token from self-supervised vision transformers, and Kaleido, an approach that enhances sample diversity by incorporating autoregressive latent priors into diffusion models.

The paper addresses a well-known challenge in generative modeling: pre-trained visual encoders—which excel at capturing semantic understanding through high-dimensional representations—do not naturally align with the requirements of generative models, which operate in compressed latent spaces to balance efficiency and quality. Prior work has tackled this mismatch through complex objectives and specialized architectures, but FAE proposes a markedly simpler solution: using a single attention layer coupled with two separate deep decoders. The framework decouples the reconstruction of the original feature space from the image generation step, allowing each component to optimize for its specific goal without forcing one representation to serve both purposes simultaneously.

The breadth of FAE's applicability—working with multiple self-supervised encoders (DINO, SigLIP) and multiple generative model families (diffusion and normalizing flows)—indicates that the approach is general rather than tailored to a specific architecture. This generality, combined with its minimal layer count, suggests the method may be accessible to practitioners seeking to incorporate high-quality pre-trained representations without engineering overhead. The reported benchmarks position FAE competitively: achieving state-of-the-art or near state-of-the-art FID on ImageNet 256×256 while also demonstrating fast learning (strong results at 80 epochs as well as 800 epochs) reflects both high quality and practical training efficiency.

No comments yet. Be the first to share your thoughts!