Stability AI, the creator of Stable Diffusion, has released SDXL model 1.0. It is an improvement to the earlier SDXL 0.9, SDXL Beta and the popular v1.5 model.
This post will go through:
- What the SDXL model is.
- Comparing images generated with the v1 and SDXL models.
- Running SDXL on AUTOMATIC1111 Web-UI.
- Running SDXL with an AUTOMATIC1111 extension.
- Running SDXL with SD.Next. (Windows)
If you want to try SDXL quickly, using it with the AUTOMATIC1111 Web-UI is the easiest way. However, it is a bit of a hassle to use the refiner in AUTOMATIC1111. SD.Next is for people who want to use the base and the refiner in one go.
What is the SDXL model?
The SDXL model is the official upgrade to the v1.5 model. The model is released as open-source software.
It is a much larger model. In the AI world, we can expect it to be better. The total number of parameters of the SDXL model is 6.6 billion, compared with 0.98 billion for the v1.5 model.
Differences between SDXL and v1.5 models
The SDXL model is, in practice, two models. You run the base model, followed by the refiner model. The base model sets the global composition. The refiner model adds finer details. (You can optionally run the base model alone.)
The language model (the module that understands your prompts) is a combination of the largest OpenClip model (ViT-G/14) and OpenAI’s proprietary CLIP ViT-L. This is a smart choice because Stable Diffusion v2 uses OpenClip alone and is hard to prompt. Bringing back OpenAI’s CLIP makes prompting easier. The prompts that work on v1.5 will have a good chance to work on SDXL.
The SDXL model has a new image size conditioning that aims to use training images smaller than 256×256. This significantly increases the training data by not discarding 39% of the images.
The U-Net, the most crucial part of the diffusion model, is now 3 times larger. Together with the larger language model, the SDXL model generates high-quality images matching the prompt closely.
The default image size of SDXL is 1024×1024. This is 4 times larger than v1.5 model’s 512×512.
Sample images from SDXL
According to Stability AI’s own study, most users prefer the images from the SDXL model over the v1.5 base model. You will find a series of images generated with the same prompts from the v1.5 and SDXL models. You can decide for yourself.
Realistic images
Let’s first compare realistic images generated using the prompt in the realistic people tutorial.
Prompt:
photo of young Caucasian woman, highlight hair, sitting outside restaurant, wearing dress, rim lighting, studio lighting, looking at the camera, dslr, ultra quality, sharp focus, tack sharp, dof, film grain, Fujifilm XT3, crystal clear, 8K UHD, highly detailed glossy eyes, high detailed skin, skin pores
Negative prompt:
disfigured, ugly, bad, immature, cartoon, anime, 3d, painting, b&w
All parameters except image sizes are kept the same for comparison. The size for the v1 models is 512×512. The size for the SDXL model is 1024×1024.
Here’s the SD 1.5 images.
Here’s the base model.
Here’s the base model with the refiner.
The refiner model improves rendering details.
Using the base v1.5 model is not doing justice to the v1 models. Most users use fine-tuned v1.5 models to generate realistic people. So I include the result using URPM, an excellent realistic model, below.
Below is another set of comparison images using a different seed value.
The SDXL base model produced a usable image in this set, although the face looks a bit too smooth for a realistic image. The refiner adds nice realistic details to the face.
Legible text
The ability to generate correct text stood out as a ground-breaking capability when I tested the SDXL Beta Model. SDXL should be at least as good.
Prompt:
A fast food restaurant on the moon with name “Moon Burger”
Negative prompt:
disfigured, ugly, bad, immature, cartoon, anime, 3d, painting, b&w
Here are the images from the SDXL base and the SDXL base with refiner.
On the other hand, the v1.5 base model fails miserably. Not only it failed to produce legible text, but it also did not generate a correct image.
Anime Style
Let’s compare the images with the Anime style.
anime, photorealistic, 1girl, collarbone, wavy hair, looking at viewer, upper body, necklace, floral print, ponytail, freckles, red hair, sunlight
disfigured, ugly, bad, immature, photo, amateur, overexposed, underexposed
Here are images from SDXL models with and without refiners.
Here are images from v1.5 and the Anything v4.5 model (fine-tuned v1.5).
The SDXL base model produces decant anime images. The images are fantastic for a base model. The refiner adds good details but seems to add some repeating artifacts. Getting to a particular style likely needs custom fine-tuned models like the v1.
Scenes
Finally, some sample images of a city with this simple prompt.
Painting of a beautiful city by Brad Rigney.
This is from v1.5 for comparison.
Download SDXL 1.0 model
You can find the SDXL base, refiner and VAE models in the following repository.
Here are the direct download links of the safetensor model files. You typically don’t need to download the VAE file unless you plan to try out different ones.
Download SDXL 1.0 refiner model
Tips on using SDXL 1.0 model
A Stability AI’s staff has shared some tips on using the SDXL 1.0 model. Here’s the summary.
- Negative prompt. Negative prompts are not as necessary in the 1.5 and 2.0 models. Many common negative terms are useless, e.g. Extra fingers.
- Keyword weight. You don’t need to use a high keyword weight like the v1 models. 1.5 is very high for the SDXL model. You may need to reduce the weights when you reuse the prompt from v1 models. Lowering a weight works better than increasing a weight.
- Safetensor. Always use the safetensor version, not the checkpoint version. It is safer and won’t execute codes on your machine.
- Refiner strength. Use a low refiner strength for the best outcome.
- Refiner. Use a noisy image to get the best out of the refiner.
- Image size. The native size is 1024×1024. SDXL supports different aspect ratios but the quality is sensitive to size. Here are the image sizes used in DreamStudio, Stability AI’s official image generator
- 21:9 – 1536 x 640
- 16:9 – 1344 x 768
- 3:2 – 1216 x 832
- 5:4 – 1152 x 896
- 1:1 – 1024 x 1024
Run SDXL model on AUTOMATIC1111
AUTOMATIC1111 Web-UI now supports the SDXL models natively. You no longer need the SDXL demo extension to run the SDXL model.
The update that supports SDXL was released on July 24, 2023. You may need to update your AUTOMATIC1111 to use the SDXL models.
You can use AUTOMATIC1111 on Google Colab, Windows, or Mac.
Download the Quick Start Guide if you are new to Stable Diffusion.
Installing SDXL 1.0 models on Google Colab
Installing the SDXL model in the Colab Notebook in the Quick Start Guide is easy. All you need to do is to select the SDXL_1 model before starting the notebook.
Installing SDXL 1.0 models on Windows or Mac
Download the SDXL base and refiner models and put them in the models/Stable-diffusion folder as usual. See the model install guide if you are new to this.
Download SDXL 1.0 refiner model
After clicking the refresh icon next to the Stable Diffusion Checkpoint dropdown menu, you should see the two SDXL models showing up in the dropdown menu.
Using SDXL base model text-to-image
Using the SDXL base model on the txt2img page is no different from using any other models. The basic steps are:
- Select the SDXL 1.0 base model in the Stable Diffusion Checkpoint dropdown menu
- Enter a prompt and, optionally, a negative prompt.
- Set image size to 1024×1024, or something close to 1024 for a different aspect ratio. (see the tips section above)
IMPORTANT: Make sure you didn’t select a VAE of a v1 model. Go to Settings > Stable Diffusion. Set SD VAE to None or Automatic.
TIPS: In Settings > User Interface > QuickSetting: Add sd_vae to add a dropdown menu for selecting VAEs next to the checkpoint dropbox.
Prompt:
1girl ,solo,high contrast, hands on the pocket, (black and white dress, looking at viewer, white and light blue theme, white and light blue background, white hair, blue eyes, full body, black footwear the light blue water on sky and white cloud and day from above, Ink painting
Negative prompt:
sketch, ugly, huge eyes, text, logo, monochrome, bad art
Size: 896 x 1152
Using the refiner model
The refiner step is done in the img2img page.
- Click Send to img2img under the output image.
2. Select the SDXL 1.0 refiner model in the Stable Diffusion Checkpoint dropdown menu.
3. Set denoising strength to 0.1-0.3 (This IS the refiner strength. Increase to add more detail).
Using preset styles for SDXL
DreamStudio, the official Stable Diffusion generator, has a list of preset styles available. They are actually implemented by adding keywords to the prompt and negative prompt. You can install the StyleSelectorXL extension to add the same list of presets styles to AUTOMATIC1111.
Installing StyleSelectorXL extension
To install the extension, navigate to the Extensions page in AUTOMATIC1111. Select the Install from URL tab. Put the following in the URL for extension’s git repository.
https://github.com/ahgsql/StyleSelectorXLPress Install. After you see the confirmation of successful installation, restart the AUTOMATIC1111 Web-UI completely.
Using SDXL style selector
You should see a new section appear on the txt2img page.
Write the prompt and the negative prompt as usual. Make sure the SDXL Styles option is enabled. Select a style other than base to apply a style.
Run SDXL model using an AUTOMATIC1111 extension
Update July 26, 2023: You no longer need this extension to run SDXL on AUTOMATIC1111.
Installing the SDXL demo extension on Windows or Mac
To install the SDXL demo extension, navigate to the Extensions page in AUTOMATIC1111. Go to the Install from URL tab. Enter the following URL in the URL for extension’s git repository field.
https://github.com/lifeisboringsoprogramming/sd-webui-xldemo-txt2img
Click Install and wait for confirmation that the extension is successfully installed.
Restart AUTOMATIC1111.
Installing the SDXL demo extension on Google Colab
Installing the SDXL demo extension on the site’s Colab notebook is easy. Enter the extension’s URL in the Extensions_from_URL field.
https://github.com/lifeisboringsoprogramming/sd-webui-xldemo-txt2img
Start the Colab Notebook as usual.
Setting up the SDXL extension
Step 1. Fill out the agreement form
Fill out this Form in HuggingFace. If you don’t see a form, it means you have already filled it out, or it is no longer required.
Step 2. Create a Huggingface token
Go to this page, create, and copy a token.
Click New Token, give it a name, and copy it.
Step 3 Enter Access token
In AUTOMATIC1111 Web-UI, navigate to the Settings page.
Go to the SDXL Demo section using the left panel selection.
Enter your Access token in the Huggingface access token field.
Select SDXL 0.9 (fp16) in the Model field.
Click Apply Settings.
Restart AUTOMATIC1111 to take effect.
For Google Colab users, you need to stop and restart the cell. Don’t disconnect the runtime.
You should see the model being downloaded in the next startup.
Now the setup is complete.
Using the SDXL demo extension
Base model
To use the SDXL base model, navigate to the SDXL Demo page in AUTOMATIC1111.
The interface is similar to the txt2img page. Enter a prompt and press Generate to generate an image.
Refiner model
To use the refiner model, select the Refiner checkbox. An image canvas will appear.
Upload the image generated from the base model to the image canvas.
Click Refine to run the refiner model.
Run SDXL model with SD.Next
AUTOMATIC1111 now supports SDXL in the stable release. However, the refiner cannot be used with the base model seamlessly. It has to be a separate step on the img2img page and requires switching the model.
For seemingly generating an image with the base and the refiner model in one go, SD.Next is a good alternative. It is built based on AUTOMATIC1111, so you will find their interface similar.
Hardware requirement
You should have a discrete Nvidia graphic card (GPU) with at least 12 GB VRAM and a Windows machine.
Install SD.Next on Windows
(You don’t need to perform steps 1 and 2 if you already have AUTOMATIC1111 running on your machine.)
Step 1: Install Python
You will need Python 3.10. Install from the Microsoft store is the easiest option.
Step 2: Install git
Git is a code repository management system. You will need it to install and update AUTOMATIC1111.
Go to this page to download the Windows version.
Open the installer. Click Install to accept the license and install the software.
Step 3: Clone SD.Next
Press the Window key (It should be on the left of the space bar on your keyboard). A search window should pop up. Type powershell. Click the Windows PowerShell Icon to launch the PowerShell App.
Type the following command and press Enter to clone SD.Next to your computer.
git clone https://github.com/vladmandic/automatic
Step 4: Run SD.Next
Open the File Explorer App.
In the address bar, go to the following location
%userprofile%automaticFind a file called webui.bat. If you don’t see the extension .bat, look for the webui file with the type Windows Batch File.
Double-click the webui.bat file to start SD.Next.
Double-click the webui.bat file to rerun it if you encounter the following error.
ImportError: cannot import name ‘deprecated’ from ‘typing_extensions’
It should correct itself the 2nd time.
Type y and press Enter when asked if you want to download the default model.
The installation is complete when you see the Local URL message, which is http://127.0.0.1:7860/ by default.
Step 5: Access the webui on a browser
Go to the local URL on the browser to start SD.Next.
http://127.0.0.1:7860/You should see a user interface very similar to the AUTOMATIC1111 Web-UI.
Test generating an image with a simple prompt (e.g. a cat) and Press Generate to make sure it is working correctly.
Download the SDXL 1.0 models
You will need to download the SDXL model to use with SD.Next. Note that there’s a malicious copy of SDXL out there, so you should download it from the official repository.
In SD.Next Web-UI, navigate to the Models page and the Huggingface tab.
In the Select model field, enter the following.
stabilityai/stable-diffusion-xl-base-1.0Press Download Model.
Verify the download is in progress in the console window.
Wait for it to complete.
Press the Download button again if you receive a read time-out error.
Repeat the download with the following in the Select model field.
stabilityai/stable-diffusion-xl-refiner-1.0Now you have completed downloading both the base and the refiner model.
Setting up SD.Next to use SDXL
SD.Next has two Stable Diffusion backends: (1) original and (2) diffusers. You must switch to the diffuers backend to use the SDXL model.
Switching to the diffusers backend
PowerShell method
You will need to start SD.Next’s webui.bat with the extra argument --backend diffuers to use the diffusers backend. You can switch back and forth between the original and the diffusers backend after you start through the GUI.
If you use the PowerShell app for the web-UI, use the following command.
cd automatic; .webui.bat --backend diffusersFile Explorer method
You can also create a new shortcut to add the extra argument for convenience.
In File Explorer, right-click the file webui.bat. Click Show more options. Click create shortcut.
A new Shortcut file is created. Rename it to webui.bat - diffusers.
Right-click on the file, select Properties. At the end of the Target field, add
--backend diffusersThere should be a space between webui.bat and --backend diffusers.
Click Apply and then OK.
From now on, you can double click the Shortcut file webui.bat - diffusers to launch SD.Next with the diffuser backend.
For convenience, you should add the refiner model dropdown menu.
Go to the Settings page, in the QuickSettings list (search quick to find it), add sd_model_refiner.
Click Apply settings and then Restart server.
After restarting, you should see the Stable Diffusion refiner dropdown menu next to the Stable Diffusion checkpoint.
Using the SDXL model
You are now ready to generate images with the SDXL model. There are two modes to generate images
- Base model alone
- Base model followed by the refiner
Base model only
Navigate to the From Text tab.
Select the SDXL base model in the Stable Diffusion checkpoint dropdown menu.
Select None in the Stable Diffuson refiner dropdown menu.
Enter a prompt.
a cat playing guitar, wearing sunglassesSet sampling steps to 30.
Set both the width and the height to 1024. This is important because the SDXL model was trained to generate 1024×1024 images.
The rest of the options are pretty standard. See the AUTOMATIC1111 guide and this article for details.
Base model + refiner
Use the base model followed by the refiner to get the best result. You will get images similar to the base model but with more fine details.
This option takes up a lot of VRAMs. Not all graphic cards can handle it.
To use the base model with the refiner, do everything in the last section except select the SDXL refiner model in the Stable Diffusion refiner dropdown menu.
Switching between original and diffusers backends
You can switch between the original and the diffusers backend on the Settings page.
Switch back to the original backend to use v1 and v2 models. Select original in Stable Diffusion backend.
Click Apply settings and Restart server.
Some notes about SDXL
Make sure to use an image size of 1024 x 1024 or similar. 512×512 doesn’t work well with SDXL.
SD.Next currently uses the diffusers backend for the SDXL model. Further optimizations using the original backend on SD.Next and AUTOMATIC1111 will be available later. That should translate to faster generation and lower VRAM requirements.
Frequently Asked Questions
Can I use SDXL on Mac?
Yes, you will need Mac with Apple Silicon M1 or M2. Make sure your AUTOMATIC1111 is up-to-date. See the installation tutorial.
Can I use ControlNet with SDXL models?
ControlNet currently only works with v1 models. SDXL is not supported.
But it appears to be in the work.
What image sizes should I use with SDXL models?
Here are the recommended image sizes for different aspect ratios.
- 21:9 – 1536 x 640
- 16:9 – 1344 x 768
- 3:2 – 1216 x 832
- 5:4 – 1152 x 896
- 1:1 – 1024 x 1024
Next step
Check out some SDXL prompts to get a quick start.
Interesting reads
Stability AI launches SDXL 0.9: A Leap Forward in AI Image Generation – Official press release of SDXL 0.9.
ANNOUNCING SDXL 1.0 — Stability AI – Official press release of SDXL 1.0
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis – Research article detailing the SDXL model.
Diffusers · vladmandic/automatic Wiki – Using Diffusers mode in SD.Next
# disable ad on single page
