Some question about Decoder finetune

[ZhikangNiu]

System Info / 系統信息

Running successfully and a question about finetune config

Who can help? / 谁可以帮助到您？

No response

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

configs/ch48_LCR_LMR_512p_DecoderFinetune.yaml

Expected behavior / 期待表现

I noticed that when you fine-tune the decoder, you will add a noise sampled from a variance = 5e-3 normal distribution. But your finetune config may not include this trick?

[Zhazhan]

Thank you for your interest in this work.
Using a Gaussian noise with 5e-3 variance to fine-tune the Decoder was not adopted in our final setup. This is because LMR can effectively replace explicit noise injection to enforce noise robustness in the Decoder, as shown in the last two rows of Table 3 in the paper. Therefore, in configs/ch48_LCR_LMR_512p_DecoderFinetune.yaml, we keep LMR enabled rather than noise injection.

[ZhikangNiu]

Thank you for your interest in this work. Using a Gaussian noise with 5e-3 variance to fine-tune the Decoder was not adopted in our final setup. This is because LMR can effectively replace explicit noise injection to enforce noise robustness in the Decoder, as shown in the last two rows of Table 3 in the paper. Therefore, in configs/ch48_LCR_LMR_512p_DecoderFinetune.yaml, we keep LMR enabled rather than noise injection.

Thank you for your response. Somehow, can I interpret LMR as a scenario with strong noise injection? From the decoder’s perspective, this would constitute a more challenging task, which might explain why it leads to better performance. In addition, I am curious about how much the benefit of fine-tuning the decoder without introducing any noise would decrease compared to using LMR or other noise-injection strategies.

[Zhazhan]

Thank you for your response. Somehow, can I interpret LMR as a scenario with strong noise injection? From the decoder’s perspective, this would constitute a more challenging task, which might explain why it leads to better performance. In addition, I am curious about how much the benefit of fine-tuning the decoder without introducing any noise would decrease compared to using LMR or other noise-injection strategies.

You are right! LMR can indeed be interpreted as a stronger noise injection. Our ablation study on noise injection is mainly conducted in the first training stage, specifically by comparing the results in the 4th (Noise Injection) and 5th (LMR) rows with those in the 3rd row (No anti-noise training for decoder) in Table 3. We did not further ablate different noise injection mechanisms in Stage 2 training. Instead, we chose to inherit the LMR setting used in Stage 1, in order to avoid making the decoder re-sensitive to noise. This could be our future work.

[ZhikangNiu]

Thank you for your response; I found it very helpful and learned a great deal from it!

[ZhikangNiu]

BTW, Have you tried applying this paradigm to images? If so, did it yield any noticeable gains?

[Zhazhan]

BTW, Have you tried applying this paradigm to images? If so, did it yield any noticeable gains?

We have done some preliminary exploration on images. The same patterns of low-frequency bias and few-mode bias still hold, but for images there are simpler ways to introduce these two biases. This is mainly because images contain much less information than videos, so the channel count of an image VAE can be significantly smaller without sacrificing reconstruction quality. As we show in Fig. 4(a) of our paper, VAEs with fewer channels naturally bear both the low-frequency bias and the few-mode bias. This may also explain why many works that aim to improve image VAEs are conducted on image VAEs with relatively large channel sizes.