Flux ControlNet model collection

Flux ControlNet

This article will compile the ControlNet model currently applicable to the Flux ecosystem

XLabs-AI/flux-controlnet-collections

XLabs-AI/flux-controlnet-collections is a ControlNet checkpoint collection provided for the FLUX.1-dev model. Developed by Black Forest Labs, this repository is designed to provide more control options for the Flux ecosystem.

Main features:

Three ControlNet models are supported:

Canny (Edge Detection)

HED (Edge Detection)

Depth (Depth map, based on Midas)

All models are trained on 1024x1024 resolution and are suitable for generating images with 1024x1024 resolution.

A v3 version is available, which is an improved, more realistic version that can be used directly in ComfyUI.

It provides custom nodes and workflows for ComfyUI, which is convenient for users to get started quickly.

Sample images and generation results are provided to show the effect of the model.

How to use:

Use the main.py script in the official repository.

Use the provided custom nodes and workflows in ComfyUI, and the official workflow is provided: https://huggingface.co/XLabs-AI/flux-controlnet-collections/tree/main/workflows

Use the Gradio demo interface.

Using the official workflow in comfyui requires the following two plugins to be installed

https://github.com/Fannovel16/comfyui_controlnet_aux

https://github.com/XLabs-AI/x-flux-comfyui

license:

These model weights are subject to the FLUX.1 [dev] non-commercial license.

Link:

Model repository: https://huggingface.co/XLabs-AI/flux-controlnet-collections

GitHub repository: Contains ComfyUI workflow, training scripts, and inference demonstration scripts.

Here is a list of ControlNet models available in the XLabs-AI/flux-controlnet-collections repository:

This project provides a powerful ControlNet model for the Flux ecosystem, allowing users to more precisely control the image generation process, especially for application scenarios that require image generation based on edge detection or depth information.

InstantX Flux Union ControlNet

InstantX Flux Union ControlNet is a versatile ControlNet model designed for the FLUX.1 development version. This model integrates multiple control modes, allowing users to more flexibly control the image generation process.

Main features:

Multiple control modes: Supports multiple control modes, including Canny edge detection, Tile (tile), depth map, blur, posture control, etc.

High performance: Most control modes achieve high effectiveness, especially Canny, Tile, depth map, blur and posture control modes.

Continuous Optimization: The development team is constantly improving the model to improve its performance and stability.

Compatibility: Fully compatible with the FLUX.1 development version basic model, it can be easily integrated into existing FLUX workflows.

Multi-control reasoning: supports the use of multiple control modes at the same time, providing users with more refined image generation control.

How to use:

Single control mode:

Loading the Union ControlNet model

Select the desired control mode (such as Canny, depth map, etc.)

Set control images and related parameters

Generate an image

Multi-control mode:

Loading the Union ControlNet model as FluxMultiControlNetModel

Set different control images and parameters for each control mode

Multiple control modes are applied simultaneously to generate images

Notes:

The current version is beta version, and may not have been fully trained yet, and some imperfect situations may be encountered during use.

Some control modes (such as grayscale control) may have lower effectiveness, and it is recommended to give priority to using highly effective modes.

The Union model may be slightly less performing on certain tasks than the dedicated single-function ControlNet model, but it provides greater flexibility and versatility.

Resource link:

Model download: InstantX/FLUX.1-dev-Controlnet-Union-alpha

Note: After downloading the model, it is recommended to rename the file to a more descriptive name, such as "flux_union_controlnet.safetensors" for future file management and identification.

This Union ControlNet model provides the FLUX ecosystem with powerful and flexible image control tools, especially for users who need to implement multiple control functions in a single model. With continuous optimization and updates, it is expected to become one of the most comprehensive and powerful ControlNet models on the FLUX platform.

InstantX Flux Canny ControlNet

In addition to the Union ControlNet model, InstantX also provides a ControlNet model specifically for Canny edge detection. This model focuses on using the Canny edge detection algorithm to control the image generation process, providing users with more precise edge control capabilities.

Main features:

Focus on Canny edge detection: This model is optimized specifically for Canny edge detection to better process and utilize edge information.

High resolution training: The model was trained in a multi-scale environment with a total number of pixels of 10241024, and 30k steps were trained using a batch size of 88.

Compatible with FLUX.1: Designed for FLUX.1 development version, seamlessly integrated into FLUX workflows.

Using bfloat16 precision: The model uses bfloat16 precision, which can improve computational efficiency while maintaining accuracy.

How to use:

Install the latest version of the Diffusers library.

Load the Flux Canny ControlNet model and the FLUX.1 base model.

Prepare the input image and apply Canny edge detection.

The detected edge is used as the control input to generate the image.

Jasperai Flux.1-dev ControlNets Series

Jasperai has developed a series of ControlNet models for Flux.1-dev, which are designed to provide more precise control for AI image generation. This series includes surface normals, depth maps and super-resolution models, providing users with a variety of creative tools.

You can use the Jasperai collection page on Hugging Face View details of these models.

1. Surface Normal ControlNet Model

Surface Normal ControlNet Model Use surface normal maps to guide image generation. This model is optimized specifically for surface normal information, which can better process and utilize geometric information on the surface of an object.

Main features:

Focus on surface normal treatment

Provide accurate geometric information on the surface of an object

Enhance the depth and reality of the image

Compatible with FLUX.1 development version

2. Depth Map ControlNet Model

The depth map ControlNet model uses depth information to control image generation. This model is specifically optimized for depth map information, which can better understand and utilize the spatial structure information of the scene.

Main features:

Focus on depth map processing

Provide information on the spatial structure of the scene

Improve the perspective and spatial sense of images

Compatible with FLUX.1 development version

3. Super Resolution ControlNet Model

Super Resolution ControlNet Model Focus on improving the quality of low-resolution images. This model can convert low-quality images into high-resolution versions, reconstructing and enhancing image details.

Main features:

Focus on image super-resolution processing

Convert low-resolution images to high-resolution versions

Reconstruct and enhance image details

Compatible with FLUX.1 development version

These models provide more precise controls for AI image generation, allowing creators to generate more realistic and detailed images. Each model is designed for specific image processing needs, providing users with diverse creative tools. Users can choose the right model according to their needs to achieve different image generation effects.