Sample-Aligned Decoders
for Latent Generative Models

1Institute of Computing Technology, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3Tsinghua University
Motivation of SAD

TL;DR. The decoder of a latent generative model is trained only on encoder latents, but at test time it decodes generated latents. SAD fine-tunes only the decoder on the generator's own samples and improves every generator we tested, at no inference cost.

Abstract

Latent generative models synthesize images by decoding generated latents, yet their decoder is trained only to reconstruct encoder latents and never sees the latents it decodes at test time, so the residual errors of the generator are rendered directly into the final image. Existing remedies retrain the generator, add cost at every sampling step, or make the decoder robust to perturbed real latents, assuming that generator errors resemble such perturbations. We find that they do not. Generated latents lie on the encoder latent manifold as closely as real ones, yet follow a different distribution. The decoder should therefore be adapted to the distribution of generated latents, which raises two challenges: (1) unpaired supervision, since a generated latent has no ground-truth image, and (2) latent-space drift, since a decoder freed from paired targets forgets the latent space it must decode.

We propose Sample-Aligned Decoders (SAD), a post-training framework that fine-tunes only the decoder of a frozen generator at no inference cost. Generated-Sample Distribution Alignment matches decoded generated latents to unpaired real images in a frozen foundation-model feature space, and Encoder Anchor Preservation keeps the decoder close to the original one on encoder latents. Our analysis shows that the alignment is effective only on generated latents, where it corrects each sample in proportion to how much more often the generator produces such latents than the encoder; on encoder latents it can only work against the anchor. SAD improves four state-of-the-art ImageNet generators across three tokenizers, e.g., FID-50K from 1.15 to 0.98 for REPA-E, complements discriminator guidance, and extends to text-to-image generation.

Method

Overview of SAD

Generated-Sample Distribution Alignment

A one-time cache of generated latents is decoded and aligned with unpaired real images in a frozen DINOv2 feature space (GAN, MMD or Fréchet discrepancy). No generated latent needs a ground-truth image.

Encoder Anchor Preservation

An L1 anchor keeps the decoder close to the original decoder on encoder latents, so reconstruction is preserved and the decoder does not overfit the alignment space.

Training takes 2,000 steps, about one GPU-hour, and never back-propagates through the generator. At inference the fine-tuned decoder simply replaces the original one, so the sampling pipeline and its cost are unchanged.

Same latent, only the decoder changes

Drag the slider: left is the original decoder D0, right is SAD.

Results on ImageNet 256×256

Comparison with state-of-the-art generators

1.15 → 0.98

FID-50K, REPA-E

1.38 → 1.23

FID-50K, LightningDiT

2.30 → 2.13

FID-50K, DiT-XL/2

−32%

FID, PixArt-α text-to-image

Decoder-side vs. generator-side correction

Discriminator guidance (DG) corrects samples at every sampling step; SAD learns the correction once in the decoder. SAD gets more than half of DG's FID gain with no extra sampling cost and 5k instead of 1.28M real images, and the two stack. DiT-XL/2, official recipe, guidance 1.5, FID-50K.

MethodExtra samplingReal imagesFID↓sFID↓IS↑
D0––2.304.51277.8
DG+20%1.28M2.014.49266.5
SAD (ours)05k2.134.25280.5
DG + SAD (ours)+20%1.28M1.864.02269.3

Comparison with other ways to change the decoder

Same DiT-XL/2 latents (50-step DPM-Solver++, 10k samples, held-out reference). Re-training the decoder or making it robust to perturbed latents does not help; aligning with real images does, and SAD gives the best held-out FDCLIP and the best reconstruction. FDDA optimizes Inception features, the network FID itself uses, and more than doubles rFID.

MethodFID↓FDCLIP↓rFID↓PSNR↑
Original decoder D013.3251.42.6323.91
Replacing or re-training the decoder
sd-vae-ft-mse13.4051.82.9124.75
Consistency Decoder13.1857.33.3423.31
Fine-tuning, ℓ1+LPIPS+GAN13.4048.67.0224.51
Robustness to perturbed latents
Noise-robust decoder13.4051.62.6724.02
Paired post-training13.1050.82.5923.91
Alignment to real images
Pixel refiner after D012.5445.32.6323.86
FDDA (Inception in the objective)12.2152.15.9121.16
SAD-GAN (ours)12.3642.42.5623.92
SAD-MMD (ours)12.6841.62.5723.92

Where you align matters

Where GSA is applied

(a) Every decoder we trained, placed by generation FID and reconstruction rFID. Aligning on generated latents (orange) lowers FID at unchanged rFID; the same loss on encoder latents (red) only breaks reconstruction.

(b) Decoded DiT samples lack high-frequency energy compared with real images. SAD recovers about half of the gap.

Per-sample correction against the density ratio

Distribution shift, not perturbation. Generated latents are not off the encoder manifold: re-encoding changes them as little as real latents (1.6% vs. 1.7%), yet a small classifier tells the two apart (AUC 0.96).

The correction follows the density ratio. Ranking every latent by that classifier, SAD changes generated latents 9× more than encoder latents, and within each set the change grows with the estimated density ratio (Spearman 0.57 and 0.31). The decoder corrects most where the generator over-represents.

Ablation (DiT-XL/2, 50-step DPM-Solver++)FID↓FDCLIP↓rFID↓PSNR↑
D013.3251.42.6323.91
w/o GSA (anchor only)13.3149.92.6523.92
GSA on E(x) instead of zgen13.5951.57.7023.90
w/o EAP13.0650.35.0621.92
SAD (GSA + EAP)12.3642.42.5623.92

Color and exposure

D0 renders DiT samples over-saturated and clips about twice as many pixels as real images. SAD brings these statistics back to those of real images, although no term in its objective targets color.

RealD0SAD
Saturation0.3230.3750.324
Colorfulness42.047.841.3
Clipped highlights4.8%9.4%5.9%
Clipped shadows1.4%4.1%1.9%

Text-to-image (PixArt-α)

COCO, 10k samples. The same recipe lowers FID by 32% and raises recall, while the CLIP score is preserved.

DecoderFID↓FDCLIP↓CLIP↑Recall↑
D034.73209.126.030.286
SAD, GAN λ=0.0123.55141.626.910.416
SAD, MMD λ=0.128.35150.426.190.343

BibTeX

@article{lei2026sad,
  title   = {Sample-Aligned Decoders for Latent Generative Models},
  author  = {Lei, Yu and Li, Yi and Gao, Hongcheng and Wei, Yipeng and Li, Liang and Su, Li and Huang, Qingming},
  journal = {arXiv preprint arXiv:2611.22458},
  year    = {2026}
}