Apple researchers propose FAE, a single-layer framework to adapt visual encoders for image generation

The paper addresses a well-known challenge in generative modeling: pre-trained visual encoders—which excel at capturing semantic understanding through high-dimensional representations—do not naturally align with the requirements of generative models, which operate in compressed latent spaces to balance efficiency and quality. Prior work has tackled this mismatch through complex objectives and specialized architectures, but FAE proposes a markedly simpler solution: using a single attention layer coupled with two separate deep decoders. The framework decouples the reconstruction of the original feature space from the image generation step, allowing each component to optimize for its specific goal without forcing one representation to serve both purposes simultaneously.

The breadth of FAE's applicability—working with multiple self-supervised encoders (DINO, SigLIP) and multiple generative model families (diffusion and normalizing flows)—indicates that the approach is general rather than tailored to a specific architecture. This generality, combined with its minimal layer count, suggests the method may be accessible to practitioners seeking to incorporate high-quality pre-trained representations without engineering overhead. The reported benchmarks position FAE competitively: achieving state-of-the-art or near state-of-the-art FID on ImageNet 256×256 while also demonstrating fast learning (strong results at 80 epochs as well as 800 epochs) reflects both high quality and practical training efficiency.