SDXL Turbo Optimization Experiments

ComfyUI and other UI powered full backend systems were super slow and I wanted to optimize it for better efficiency and real time performane for a local running LLM based ppt generator application I was building

I wrote a custom script to benchmark different stages of the image generation while trying out different configurations and this is just me writing down my finding for future reference primarily

My Specs and Configuration:

Ryzen 7 5800X | 32 GB RAM | RTX 3060 Ti

Windows 11 WSL Ubuntu, python 3.10

Comfy UI SDXL Turbo Generation Speed averages at 2.5 second per image (without prompt caching)

My Test Criterias distinguishes Total Time, Load Time for any modules, Init Gen Time, Avg Gen Time

For Avg Time calculation I test with 1 + 4 prompts

I am not optimizing for memory footprint, just performance

Here is my Github for reference : https://github.com/Govind-S-B/sdxl-turbo-optimization-experiments

I also post some experiments I do over at my twitter : https://twitter.com/violetto96

Basic Diffuser Performance :

The default example script from sdxl turbo repo was used here, with multiple prompts being iterated

Batch Prompt Processing :

Instead of iterating through each prompt, passed the entire list for batch processing to the pipeline

Its suprising to see a decline in performance while using the built in batch processing methond than iterating the pipe

Upcase VAE Precision

Decrease in performance compared to base diffuser performance

source : https://huggingface.co/docs/diffusers/using-diffusers/sdxl_turbo#speed-up-sdxl-turbo-even-more

VAE 16fp Optimization :

Gives Promising Speed (2x) with no noticeable quality loss

source : https://huggingface.co/docs/diffusers/using-diffusers/sdxl_turbo#speed-up-sdxl-turbo-even-more

pipe.vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

VAE : Tiny Auto Encoder

Incredible Speed, Its said to be quality reducing but I couldnt notice

source : https://github.com/huggingface/blog/blob/main/simple_sdxl_optimizations.md#tiny-autoencoder

pipe.vae = AutoencoderTiny.from_pretrained("madebyollin/taesdxl", torch_dtype=torch.float16)

Doing all subsequent tests with Tiny Auto Encoder since the vae is pretty much optimized and we will not need to swap that out, for all following performance comparisons use the Tiny Auto Encoder benchmark as reference point now

Compile UNet

Massive Decrease in init generation and 2x slower in avg gen time

source : https://huggingface.co/docs/diffusers/using-diffusers/sdxl_turbo#speed-up-sdxl-turbo-even-more

pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)

CPU Offloading

Decrease in performance

source : https://github.com/huggingface/blog/blob/main/simple_sdxl_optimizations.md#model-cpu-offloading

General rule of thumb that if you can fit the model in memory, offloading is only going to cost you performance

VAE Slicing

No Impact or Slight Decrease in performance , hard to gauge.

source : https://github.com/huggingface/blog/blob/main/simple_sdxl_optimizations.md#slicing

MORE TESTING CONCLUSIONS

I tried all the optimizations listed in https://huggingface.co/docs/diffusers/optimization/opt_overview

Including Xformers , Token Merging , Offloading . But none of them beats the base Tiny Encoder Optimization Benchmarks

I think it indicates some bottleneck deep within python thats causing the performance issue now and I have hit a bottleneck in what I could optimize,if anyone else wants to try out the optimizations in a different language like Rust I think that would be the way forward

Additional Notes :

• I couldnt try out https://github.com/huggingface/blog/blob/main/simple_sdxl_optimizations.md#caching-computations since i couldnt figure out how to get the tokenizers and encoders for the model
• Couldnt try out tracing UNet https://huggingface.co/docs/diffusers/optimization/memory#tracing