Duplicate from ControlNet-1-1-preview/control_v11p_sd15s2_lineart_anime

.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,152 @@
+---
+license: openrail
+base_model: runwayml/stable-diffusion-v1-5
+tags:
+- art
+- controlnet
+- stable-diffusion
+duplicated_from: ControlNet-1-1-preview/control_v11p_sd15s2_lineart_anime
+---
+
+# Controlnet - v1.1 - *lineart_anime Version*
+
+**Controlnet v1.1** is the successor model of [Controlnet v1.0](https://huggingface.co/lllyasviel/ControlNet)
+and was released in [lllyasviel/ControlNet-v1-1](https://huggingface.co/lllyasviel/ControlNet-v1-1) by [Lvmin Zhang](https://huggingface.co/lllyasviel).
+
+This checkpoint is a conversion of [the original checkpoint](https://huggingface.co/lllyasviel/ControlNet-v1-1/blob/main/control_v11p_sd15s2_lineart_anime.pth) into `diffusers` format.
+It can be used in combination with **Stable Diffusion**, such as [runwayml/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5).
+
+
+For more details, please also have a look at the [🧨 Diffusers docs](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/controlnet).
+
+
+ControlNet is a neural network structure to control diffusion models by adding extra conditions. 
+
+![img](./sd.png)
+
+This checkpoint corresponds to the ControlNet conditioned on **lineart_anime images**.
+
+## Model Details
+- **Developed by:** Lvmin Zhang, Maneesh Agrawala
+- **Model type:** Diffusion-based text-to-image generation model
+- **Language(s):** English
+- **License:** [The CreativeML OpenRAIL M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license) is an [Open RAIL M license](https://www.licenses.ai/blog/2022/8/18/naming-convention-of-responsible-ai-licenses), adapted from the work that [BigScience](https://bigscience.huggingface.co/) and [the RAIL Initiative](https://www.licenses.ai/) are jointly carrying in the area of responsible AI licensing. See also [the article about the BLOOM Open RAIL license](https://bigscience.huggingface.co/blog/the-bigscience-rail-license) on which our license is based.
+- **Resources for more information:** [GitHub Repository](https://github.com/lllyasviel/ControlNet), [Paper](https://arxiv.org/abs/2302.05543).
+- **Cite as:**
+
+  @misc{zhang2023adding,
+    title={Adding Conditional Control to Text-to-Image Diffusion Models}, 
+    author={Lvmin Zhang and Maneesh Agrawala},
+    year={2023},
+    eprint={2302.05543},
+    archivePrefix={arXiv},
+    primaryClass={cs.CV}
+  }
+
+## Introduction
+
+Controlnet was proposed in [*Adding Conditional Control to Text-to-Image Diffusion Models*](https://arxiv.org/abs/2302.05543) by 
+Lvmin Zhang, Maneesh Agrawala.
+
+The abstract reads as follows:
+
+*We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions. 
+The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (< 50k). 
+Moreover, training a ControlNet is as fast as fine-tuning a diffusion model, and the model can be trained on a personal devices. 
+Alternatively, if powerful computation clusters are available, the model can scale to large amounts (millions to billions) of data. 
+We report that large diffusion models like Stable Diffusion can be augmented with ControlNets to enable conditional inputs like edge maps, segmentation maps, keypoints, etc. 
+This may enrich the methods to control large diffusion models and further facilitate related applications.*
+
+## Example
+
+It is recommended to use the checkpoint with [Stable Diffusion v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) as the checkpoint 
+has been trained on it.
+Experimentally, the checkpoint can be used with other diffusion models such as dreamboothed stable diffusion.
+
+**Note**: If you want to process an image to create the auxiliary conditioning, external dependencies are required as shown below:
+
+1. Install https://github.com/patrickvonplaten/controlnet_aux
+
+```sh
+$ pip install controlnet_aux==0.3.0
+```
+
+2. Let's install `diffusers` and related packages:
+
+```
+$ pip install diffusers transformers accelerate
+```
+
+3. Run code:
+
+```python
+import torch
+import os
+from huggingface_hub import HfApi
+from pathlib import Path
+from diffusers.utils import load_image
+from PIL import Image
+import numpy as np
+from controlnet_aux import LineartAnimeDetector
+from transformers import CLIPTextModel
+
+from diffusers import (
+    ControlNetModel,
+    StableDiffusionControlNetPipeline,
+    UniPCMultistepScheduler,
+)
+
+checkpoint = "lllyasviel/control_v11p_sd15s2_lineart_anime"
+
+image = load_image(
+    "https://huggingface.co/lllyasviel/control_v11p_sd15s2_lineart_anime/resolve/main/images/input.png"
+)
+image = image.resize((512, 512))
+
+prompt = "A warrior girl in the jungle"
+processor = LineartAnimeDetector.from_pretrained("lllyasviel/Annotators")
+
+control_image = processor(image)
+control_image.save("./images/control.png")
+
+# we skip one layer of the encoder
+text_encoder = CLIPTextModel.from_pretrained("runwayml/stable-diffusion-v1-5", subfolder="text_encoder", num_hidden_layers=11, torch_dtype=torch.float16)
+
+controlnet = ControlNetModel.from_pretrained(checkpoint, torch_dtype=torch.float16)
+
+pipe = StableDiffusionControlNetPipeline.from_pretrained(
+    "runwayml/stable-diffusion-v1-5", text_encoder=text_encoder, controlnet=controlnet, torch_dtype=torch.float16
+)
+
+pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
+pipe.enable_model_cpu_offload()
+
+generator = torch.manual_seed(0)
+image = pipe(prompt, num_inference_steps=30, generator=generator, image=control_image).images[0]
+
+image.save('images/image_out.png')
+
+```
+
+![bird](./images/input.png)
+
+![bird_canny](./images/control.png)
+
+![bird_canny_out](./images/image_out.png)
+
+## Other released checkpoints v1-1
+
+The authors released 14 different checkpoints, each trained with [Stable Diffusion v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) 
+on a different type of conditioning:
+
+| Model Name | Control Image Overview| Control Image Example | Generated Image Example |
+|---|---|---|---|
+TODO
+
+### Training
+
+TODO
+
+### Blog post
+
+For more information, please also have a look at the [Diffusers ControlNet Blog Post](https://huggingface.co/blog/controlnet).

config.json

@@ -0,0 +1,42 @@
+{
+  "_class_name": "ControlNetModel",
+  "_diffusers_version": "0.16.0.dev0",
+  "_name_or_path": "/home/patrick/controlnet_v1_1/control_v11p_sd15s2_lineart_anime",
+  "act_fn": "silu",
+  "attention_head_dim": 8,
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "class_embed_type": null,
+  "conditioning_embedding_out_channels": [
+    16,
+    32,
+    96,
+    256
+  ],
+  "controlnet_conditioning_channel_order": "rgb",
+  "cross_attention_dim": 768,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_time_scale_shift": "default",
+  "upcast_attention": false,
+  "use_linear_projection": false
+}

control_net_lineart_anime.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python3
+import torch
+import os
+from huggingface_hub import HfApi
+from pathlib import Path
+from diffusers.utils import load_image
+from controlnet_aux import LineartAnimeDetector
+from transformers import CLIPTextModel
+
+from diffusers import (
+    ControlNetModel,
+    StableDiffusionControlNetPipeline,
+    UniPCMultistepScheduler,
+)
+import sys
+
+checkpoint = sys.argv[1]
+
+url = "https://static.wikia.nocookie.net/unanything/images/a/a0/Unnamed.png/revision/latest/scale-to-width-down/350?cb=20230326002111"
+image = load_image(url)
+
+prompt = "warrior girl"
+
+processor = LineartAnimeDetector.from_pretrained("lllyasviel/Annotators")
+image = processor(image)
+image.save("/home/patrick/images/check.png")
+
+text_encoder = CLIPTextModel.from_pretrained("Linaqruf/anything-v3.0", subfolder="text_encoder", num_hidden_layers=11, torch_dtype=torch.float16)
+
+controlnet = ControlNetModel.from_pretrained(checkpoint, torch_dtype=torch.float16)
+pipe = StableDiffusionControlNetPipeline.from_pretrained(
+    "runwayml/stable-diffusion-v1-5", controlnet=controlnet, text_encoder=text_encoder, torch_dtype=torch.float16
+)
+
+pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
+pipe.enable_model_cpu_offload()
+
+generator = torch.manual_seed(33)
+out_image = pipe(prompt, num_inference_steps=25, generator=generator, image=image).images[0]
+
+path = os.path.join(Path.home(), "images", "aa.png")
+out_image.save(path)
+
+api = HfApi()
+
+api.upload_file(
+    path_or_fileobj=path,
+    path_in_repo=path.split("/")[-1],
+    repo_id="patrickvonplaten/images",
+    repo_type="dataset",
+)
+print("https://huggingface.co/datasets/patrickvonplaten/images/blob/main/aa.png")

diffusion_pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a93db77e1cac8d298d4313c183d942ffbb51c61e77104d7f17ef70217638d09b
+size 1445254969

diffusion_pytorch_model.fp16.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e628ab978db5d0094f3ab9a8608b44a8715e3bfe4c4ef425cb244d0f8d70f25
+size 722698343

diffusion_pytorch_model.fp16.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9619372316fb8ade82353be0c2b7821fa1da60ae3842d8c99afacf1d45ff73f
+size 722598642

diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d55de2de4cd8813b88e0fb603e52616c9e5ee6cb019e28f0f64ef370e363fff
+size 1445157124