Some problem when fine-tuning Infinity-2B + VAE_d32_reg models

[Bamboos2003]

google/flan-t5-large, xl, xxl has 780M, 3B, 11B params.
The Infinity uses flan-t5-xl of 3B as vanilla. But My GPU is a single RTX4090 of 24 VRAM, which get OOM error when encoding the texts.
I want to use google/flan-t5-large as text encoder.
But there is an error that the tensor's size doesn't fit the Infinity-2B model's state_dict.

If Infinity-2B model is only compatible with flan-t5-xl and xxl of 2048 dimensions output?
How can I use Infinity-2B model and google/flan-t5-large at the same time?