I thought for fast text-to-image synthesizer you would need a GAN instead of a diffusion model. GAN models are much faster. Though apparently they aren't quite competitive with diffusion models in terms of quality. See

https://arxiv.org/abs/2301.09515