On the 22nd of August, Stability.ai founder Emad Mostaque announced the release of Stable Diffusion - an AI generative art model that has not only superior capabilities to the likes of DALL·E 2 but is also available as an open-source project. In those 2 weeks since its release, people have abandoned their personal endeavors and projects to give Stable Diffusion their full attention.
I was already quite excited when OpenAI announced DALL·E 2, and I was also fortunate enough to get early access. But having toyed around with Stable Diffusion for the last couple of days, I can say that DALL·E 2 doesn't come even close to what Stable Diffusion brings to the table.
And the fact that it is open-source also makes it much more accessible. In just 2 weeks, sites like Lexica have archived over 10 million AI-generated images. I also fully expect that developers will make steady strides to integrate Stable Diffusion with the most popular graphic design tools, such as Figma, Sketch, and others. The ability to generate high-quality art on the go is unprecedented.
"Kerala village, sharp focus, wide shot, trending on artstation, masterpiece, by greg rutkowski, by ross tran, by fenghua zhong, octane, soft render, oil on canvas, colorful, cinematic, environmental concept art"
The purpose of this article is to list all the interesting and relevant tutorials, resources, and tools to help you get up to speed with Stable Diffusion quickly. I do believe over the coming months we're going to see a massive influx of projects that specialize in extracting the most potential out of Stable Diffusion, and I will do my best to keep this article updated accordingly.
- 👉 Tutorials - this section is focused heavily on topics such as "How to use Stable Diffusion?".
- 💻 Resources - this section is focused on queries such as "What is Stable Diffusion?".
- 🚀 Tools - this section is based on tools that let you use Stable Diffusion yourself.
So without further ado - let's start with the basics.
Resources & Information
One of the first questions a lot of people have about Stable Diffusion is the license that this model is published under, and whether the generated art is free to use for personal and commercial projects.
The license Stable Diffusion is using is CreativeML Open RAIL-M, and can be read in full over at Hugging Face. In short, "Open Responsible AI Licenses (Open RAIL) are licenses designed to permit free and open access, re-use, and downstream distribution of derivatives of AI artifacts as long as the behavioral-use restrictions always apply (including to derivative works).".
A more detailed explanation for this license is available on this BigScience page.
What images were used to train the Stable Diffusion model?
AI modeling is the means of creating and training Machine Learning algorithms for a specific purpose. In this case, the purpose of generating images from user prompts.
If you're curious as to which images Stable Diffusion used - Andy Baio together with Simon Willison did a thorough analysis of over 12 million images (from a total of 2.3 billion) that were used to train the Stable Diffusion model.
Here are some of the key takeaways:
- The datasets which were used to train Stable Diffusion were the ones put together by LAION.
- Out of the 12 million images they sampled, 47% of the entire sample size came from 100 domains, with Pinterest yielding as much as 8.5% of the entire dataset. Other top sources included WordPress.com, Blogspot, Flickr, DeviantArt, and Wikimedia.
- Stable Diffusion doesn't restrict the use of generating art from people's names (be it celebrities or otherwise).
It will be interesting to see how the model evolves, and whether companies will be willing to contribute their own media to help Stable Diffusion grow.
Where to find Stable Diffusion examples & prompts?
One of the ways in which Stable Diffusion differs from the likes of DALL·E is that to get the most out of Stable Diffusion you have to learn about its modifiers. One modifier, in particular, is called the seed - whenever you generate an image with Stable Diffusion, that image will be assigned a seed, which can also be understood as the general composition of that image. So, if you enjoyed a particular image and wish to replicate its style (or at least as close as possible to it) - you can use seeds.
The best platform for finding example images and the prompts that were used to generate those images is Lexica, which archives over 10 million sample artworks. Each artwork includes its full prompt, but also the seed number which you can then reuse yourself.
Tools & Software
If you've seen or been captivated by the art created with Stable Diffusion, you might be wondering whether you can try it out for yourself. And the answer is yes, there are multiple ways to try Stable Diffusion for free, including doing so from the browser or your own machine.
The official way to do so is to use the DreamStudio platform.
Anyone can register for free, and new accounts get a complimentary 200 free tokens. These tokens are sufficient for 200 generations as long as you don't increase complexity and don't change the height and width past the 512x512 default setting. But if you do increase the complexity, you're probably going to run out of your tokens rather quickly.
How to run Stable Diffusion on your own machine?
At this time, the most popular solution for running Stable Diffusion locally is the Stable Diffusion Web UI repo available on GitHub. Based on the Gradio GUI - this is as close as it gets to the actual DreamStudio interface, and you can wave goodbye to any kind of limitations.
What are the PC requirements for Stable Diffusion?
- 4GB (more is preferred) VRAM GPU (Official support for Nvidia only!)
- AMD users check here
- M1/M2 users check here and here also
Keep in mind that to use the Web UI repo, you will need to download the model yourself from Hugging Face. Make sure you fully read the Installation guide (Windows) to set it up properly. For Linux, check this guide. And you can also get it up and running on Google Colab - guide here.
How big is the Stable Diffusion model?
As I mention above, you have to download the Stable Diffusion model yourself, and the link to do so can be found here. You will need to create an account on Hugging Face, and afterward, accept the license terms for the model before you're able to view and download its files.
One of the questions people have is, "How come the model is only 4GB in size even though it has been made from over 2 billion images?".
And the best answer to this question comes from a Hacker News user juliendorra ⟶
That’s the interesting part: all the images generated are derived from a less than 4gb model (the trained weights of the neural network).
So in a way, hundreds of billions of possible images are all stored in the model (each a vector in multidimensional latent space) and turned into pixels on demand (drived by the language model that knows how to turn words into a vector in this space)
As it’s deterministic (given the exact same request parameters, random seed included, you get the exact same image) it’s a form of compression (or at least encoding decoding) too: I could send you the parameters for 1 million images that you would be able to recreate on your side, just as a relatively small text file.
Tutorials & Guides
The following section is dedicated entirely to tutorials and guides to help you extract the most juice out of your Stable Diffusion prompts. As I said, I will do my best to keep this updated as more guides become available and a better understanding of the model is gained.
Ultimate Beginner's Guide
Arman Chaudhry published a compact Google Docs presentation on the essentials of SD.
This guide covers all of the modifiers that SD supports but also recommends the best practices for width/height settings, as well as common mistakes to avoid.
Stable Diffusion Akashic Records
If you're looking to do a deep dive (or need references for research) - the SD Akashic Records repository has an astounding amount of resources for you to study.
You'll find everything from keyword usage, to prompt optimization, to style guides. There are also mentions of several tools, outside of those already mentioned in this article.
Stable Diffusion Prompt Cheatsheet
If you're looking for inspiration to apply custom styles and effects to your prompts, check out this blog post from Moritz. It covers prompt additions for concepts like 2D & 3D art, details, lighting, colors, and environments.
Top comments (1)
I installed it this weekend on my 4year old gaming laptop, and I am very impressed. i am impressed and also see my limitations. i will definetly keep playing with it. I wonder If i can make a game out if it, tell stories because.
I myself can not draw, i am dreaming that this tool actually will democratise art for people who do not have talent at drawing.
Stay tuned: I think this week I will share my first impressions and learnings with this AI.