TL;DR. The decoder of a latent generative model is trained only on encoder latents, but at test time it decodes generated latents. SAD fine-tunes only the decoder on the generator's own samples and improves every generator we tested, at no inference cost.
Latent generative models synthesize images by decoding generated latents, yet their decoder is trained only to reconstruct encoder latents and never sees the latents it decodes at test time, so the residual errors of the generator are rendered directly into the final image. Existing remedies retrain the generator, add cost at every sampling step, or make the decoder robust to perturbed real latents, assuming that generator errors resemble such perturbations. We find that they do not. Generated latents lie on the encoder latent manifold as closely as real ones, yet follow a different distribution. The decoder should therefore be adapted to the distribution of generated latents, which raises two challenges: (1) unpaired supervision, since a generated latent has no ground-truth image, and (2) latent-space drift, since a decoder freed from paired targets forgets the latent space it must decode.
We propose Sample-Aligned Decoders (SAD), a post-training framework that fine-tunes only the decoder of a frozen generator at no inference cost. Generated-Sample Distribution Alignment matches decoded generated latents to unpaired real images in a frozen foundation-model feature space, and Encoder Anchor Preservation keeps the decoder close to the original one on encoder latents. Our analysis shows that the alignment is effective only on generated latents, where it corrects each sample in proportion to how much more often the generator produces such latents than the encoder; on encoder latents it can only work against the anchor. SAD improves four state-of-the-art ImageNet generators across three tokenizers, e.g., FID-50K from 1.15 to 0.98 for REPA-E, complements discriminator guidance, and extends to text-to-image generation.
A one-time cache of generated latents is decoded and aligned with unpaired real images in a frozen DINOv2 feature space (GAN, MMD or Fréchet discrepancy). No generated latent needs a ground-truth image.
An L1 anchor keeps the decoder close to the original decoder on encoder latents, so reconstruction is preserved and the decoder does not overfit the alignment space.
Training takes 2,000 steps, about one GPU-hour, and never back-propagates through the generator. At inference the fine-tuned decoder simply replaces the original one, so the sampling pipeline and its cost are unchanged.
Drag the slider: left is the original decoder D0, right is SAD.
1.15 → 0.98
FID-50K, REPA-E
1.38 → 1.23
FID-50K, LightningDiT
2.30 → 2.13
FID-50K, DiT-XL/2
−32%
FID, PixArt-α text-to-image
Discriminator guidance (DG) corrects samples at every sampling step; SAD learns the correction once in the decoder. SAD gets more than half of DG's FID gain with no extra sampling cost and 5k instead of 1.28M real images, and the two stack. DiT-XL/2, official recipe, guidance 1.5, FID-50K.
| Method | Extra sampling | Real images | FID↓ | sFID↓ | IS↑ |
|---|---|---|---|---|---|
| D0 | – | – | 2.30 | 4.51 | 277.8 |
| DG | +20% | 1.28M | 2.01 | 4.49 | 266.5 |
| SAD (ours) | 0 | 5k | 2.13 | 4.25 | 280.5 |
| DG + SAD (ours) | +20% | 1.28M | 1.86 | 4.02 | 269.3 |
Same DiT-XL/2 latents (50-step DPM-Solver++, 10k samples, held-out reference). Re-training the decoder or making it robust to perturbed latents does not help; aligning with real images does, and SAD gives the best held-out FDCLIP and the best reconstruction. FDDA optimizes Inception features, the network FID itself uses, and more than doubles rFID.
| Method | FID↓ | FDCLIP↓ | rFID↓ | PSNR↑ |
|---|---|---|---|---|
| Original decoder D0 | 13.32 | 51.4 | 2.63 | 23.91 |
| Replacing or re-training the decoder | ||||
| sd-vae-ft-mse | 13.40 | 51.8 | 2.91 | 24.75 |
| Consistency Decoder | 13.18 | 57.3 | 3.34 | 23.31 |
| Fine-tuning, ℓ1+LPIPS+GAN | 13.40 | 48.6 | 7.02 | 24.51 |
| Robustness to perturbed latents | ||||
| Noise-robust decoder | 13.40 | 51.6 | 2.67 | 24.02 |
| Paired post-training | 13.10 | 50.8 | 2.59 | 23.91 |
| Alignment to real images | ||||
| Pixel refiner after D0 | 12.54 | 45.3 | 2.63 | 23.86 |
| FDDA (Inception in the objective) | 12.21 | 52.1 | 5.91 | 21.16 |
| SAD-GAN (ours) | 12.36 | 42.4 | 2.56 | 23.92 |
| SAD-MMD (ours) | 12.68 | 41.6 | 2.57 | 23.92 |

(a) Every decoder we trained, placed by generation FID and reconstruction rFID. Aligning on generated latents (orange) lowers FID at unchanged rFID; the same loss on encoder latents (red) only breaks reconstruction.
(b) Decoded DiT samples lack high-frequency energy compared with real images. SAD recovers about half of the gap.

Distribution shift, not perturbation. Generated latents are not off the encoder manifold: re-encoding changes them as little as real latents (1.6% vs. 1.7%), yet a small classifier tells the two apart (AUC 0.96).
The correction follows the density ratio. Ranking every latent by that classifier, SAD changes generated latents 9× more than encoder latents, and within each set the change grows with the estimated density ratio (Spearman 0.57 and 0.31). The decoder corrects most where the generator over-represents.
| Ablation (DiT-XL/2, 50-step DPM-Solver++) | FID↓ | FDCLIP↓ | rFID↓ | PSNR↑ |
|---|---|---|---|---|
| D0 | 13.32 | 51.4 | 2.63 | 23.91 |
| w/o GSA (anchor only) | 13.31 | 49.9 | 2.65 | 23.92 |
| GSA on E(x) instead of zgen | 13.59 | 51.5 | 7.70 | 23.90 |
| w/o EAP | 13.06 | 50.3 | 5.06 | 21.92 |
| SAD (GSA + EAP) | 12.36 | 42.4 | 2.56 | 23.92 |
D0 renders DiT samples over-saturated and clips about twice as many pixels as real images. SAD brings these statistics back to those of real images, although no term in its objective targets color.
| Real | D0 | SAD | |
|---|---|---|---|
| Saturation | 0.323 | 0.375 | 0.324 |
| Colorfulness | 42.0 | 47.8 | 41.3 |
| Clipped highlights | 4.8% | 9.4% | 5.9% |
| Clipped shadows | 1.4% | 4.1% | 1.9% |
COCO, 10k samples. The same recipe lowers FID by 32% and raises recall, while the CLIP score is preserved.
| Decoder | FID↓ | FDCLIP↓ | CLIP↑ | Recall↑ |
|---|---|---|---|---|
| D0 | 34.73 | 209.1 | 26.03 | 0.286 |
| SAD, GAN λ=0.01 | 23.55 | 141.6 | 26.91 | 0.416 |
| SAD, MMD λ=0.1 | 28.35 | 150.4 | 26.19 | 0.343 |
@article{lei2026sad,
title = {Sample-Aligned Decoders for Latent Generative Models},
author = {Lei, Yu and Li, Yi and Gao, Hongcheng and Wei, Yipeng and Li, Liang and Su, Li and Huang, Qingming},
journal = {arXiv preprint arXiv:2611.22458},
year = {2026}
}