A trend that has been picking up steam lately in the world of cutting-edge artificial intelligence (AI) research involves mixing and matching elements from different model architectures. Take a little of this, a little of that, and… voilà, a new architecture that solves an existing problem in a more efficient manner. And why not? Many major algorithmic advances have been made in the past few years, so why not take the best pieces and repurpose them for the biggest advantage? It sure beats racking your brain trying to invent something completely new.
We recently reported on one such instance of architecture mixing with Inception Labs’ Mercury models that incorporate diffusers — elements normally found in text-to-image generators — to speed up traditional autoregressive large language models (LLMs). And now a team of researchers at MIT and NVIDIA has just reported on their work in which they incorporate an autoregressive model into a diffusion-based image generator to speed it up. Huh? At first glance, it sounds like these two innovations are at odds with one another — but it all comes down to the specifics of exactly how models are combined.
An overview of the system architecture (📷: H. Tang et al.)
The new system, known as Hybrid Autoregressive Transformer (HART), combines the strengths of two of the most dominant model types used in generative AI today. Autoregressive models, like those used in LLMs, generate images quickly by predicting pixels in sequence. However, they often lack the fine detail needed for high-quality images. On the other hand, diffusion models create much more detailed images through an iterative denoising process but are computationally expensive and slow.
The team’s innovation lies in the way that they combined these two models. They leveraged an autoregressive model for generating the initial broad structure of the image, followed by a small diffusion model that refines the fine details. This allows HART to generate images at speeds nearly nine times faster than traditional diffusion models while maintaining — or even improving — image quality.
This architecture makes the new model highly efficient. Typical diffusion models require multiple iterations — sometimes 30 or more — to refine an image. HART’s diffusion component only needs about eight steps since most of the heavy lifting has already been done by the autoregressive model. This results in lower computational costs, making HART capable of running on standard commercial laptops or even smartphones in many cases.
A collection of images generated by HART (📷: H. Tang et al.)
Compared to existing state-of-the-art diffusion models, HART offers a 31% reduction in computational requirements while still matching — or outperforming — them in key metrics like Fréchet Inception Distance, which measures image quality. The model also integrates more easily with multimodal AI systems, which combine text and images, making it well-suited for next-generation AI applications.
The team believes that HART could have applications beyond just image generation. Its speed and efficiency could make it useful for training AI-powered robots in simulated environments, allowing them to process visual data faster and more accurately. Similarly, video game designers could use HART to generate detailed landscapes and characters in a fraction of the time required by traditional methods.
Looking ahead, the researchers hope to extend the HART framework to also work with video and audio. Given its ability to merge speed with quality, HART could play a role in advancing AI models that generate entire multimedia experiences in real time.
