Jerry Davos

[Update] [v2.1] Animate ControlNet Animation - LCM [Early Access]

Added 2023-12-21 17:50:19 +0000 UTC

VIDEO TUTORIAL : https://youtu.be/qczh3caLZ8o

The Animation Workflow is divided into 4 parts :

Part 1 - ControlNet Passes Export
Part 2 - Animation Raw - LCM
Part 3 - AnimateDiff Refiner - LCM
Part 4 - AnimateDiff Face Fix - LCM

[PART 1] - ControlNet Passes Export

All the necessary control passes are extracted with this workflow, it serves as a base dough for making the initial raw animation frames.

Export At Least 2 Passes for Raw Animation

Instruction

1) "Copy as Path" - Right Click on your Video inside explorer and click copy as path and paste it inside the Input Video Path node. It should NOT be in quotes. Eg: F:\Test\video.mp4
2) Copy Address - Copy the address of the folder you want to save the Passes and paste it in the Output path node. It should NOT be in quotes. Eg: F:\Test
3) Select the Passes you want to extract. Enable = 1, Disable = 0. Edit their properties if you want.
4) [Optional] If you have a high-resolution video like 4k, you might want to downscale it to 1k to get optimal results for rendering. Enter multiplier in "scale_by" input.
5) [optional] You can also custom resize it. Enable it by entering value as 1 and use custom dimensions.
6) Save Source Images is enabled by default for original reference and sequencing.

if your video is longer than 20 second, you may encounter "Out of Memory" errors, then use the batch workflow.

Batch Range: Specify how much frames your pc can handle in one go. (eg: 400 Images for 1280x720 size is safe for 8GB vram, 200-300 frames for 1920x1080). If your save node crashes use lower.
Skip Frames: After Every Batch skip the total rendered frames.

How to use Batch workflow Example:

If you have a 30 second video of 1280x720 with 30 fps then your total frames will be 30x30 = 900 frames

Then

Batch 1: Range - 400 | Skip Frames - 0
Batch 2: Range - 400 | Skip Frames - 400
Batch 3: Range - 100 | Skip Frames - 800

Total rendered frames: 900

ControlNet Usage Tips

For Face Closeups - Use Line Art and soft edge.
For Half Body Shots - Use Line Art with Soft edge or Open Pose.
For Full body shots - Use Open Pose with line Art or Soft edge
For Architecture or landscape - Use Depth with Soft edge, line art or Canny.
For Flat Logos, or Typos - Use Normal with Line Art, Soft edge.

At least 2 Passes are required to use raw animation workflow and get a quality render.

Results
It will export your enabled passes in the specified folder

________________________________________________________________________

[PART 2] Animation Raw

This is the heart of the workflow, main raw animation will be rendered here, where it be realistic, anime, cartoon or abstract.

Instruction

1) Choose Model checkpoint you want to use
2) Enter Width and Height of your Output Image
3) Choose Positive Prompts you want to see in the output
4) Choose Loras, (You have to experiment which looks good or not)
5) Paste the Passes Directories in ControlNets (1 and 2) Loaders and set their models and weights (detailed note inside workflow)
6) Paste The Directory where you want to save the Images in the Output Save Directory Node
7) Select Batch Range (Number of images you want to render in one Queue)
8) Choose LCM, Highres and AD Motion Settings, Default is good but experimental (detailed note inside workflow)
9) After every batch you can increase the batch naming for organizing the batches.
10) After every queue increase the skip frames to the Total number of images rendered already.
11) After All the batches are rendered it's Ready for #3 Refiner workflow

________________________________________________________________________

[Part 3] Refiner

This workflow further enhances the raw animations, add details, upscale and refines it.

Instruction

After you have rendered the raw Images from #2 Main Animation Raw file,

1) Enter the Paths in Purple Directory Nodes of the Raw Images from #2
2) Enter the Output path for saving the refined images.
3) Enter Batch Range
4) Overlapping Frames - 0 is Default (Put 5 or 10 for cross fade technique between batches)
5) Skip Frame is 0 by default (Increase it after every batch)
6) Batch Number Naming, increase it by 1 after every batch for organizing.
7) Refiner Upscale - Upscale value is 1.2 by default (Use 1.2 - 1.8 for best result for SD models)
8) See the Red note below for changing the refiner settings.
9) Enter Prompts for more accuracy.
10) Choose Model checkpoint you want to use for refiner.
11) Loras work as usual (Some of not compatible with animatediff, test it first with 10 frames)
12) After Rendering all the frames then Open the #4 AnimateDIff Face Fix if your faces are not looking good.

You can skip 12th step If faces are looking good or does not have faces in the video.

________________________________________________________________________

[Part 4] Face Fix

In this workflow, faces of the refined workflow will be enhanced. As The refiner work on all over the image, the faces might be not up to the mark, in this workflow "Only faces" are worked upon.

The Current Version 2.1 face fix workflow performs better than v2.0

Instruction

After you have refined the Images in #3 Refiner, You are ready for Face Fix workflow

1) Enter the Paths in Purple Directory Nodes of the Refined Images from #3
2) Enter the Output path for saving them
3) Enter Batch Range for face fix.
4) Skip Frame is 0 by default (Ignore if you put all images in batch range)
5) Use Batch Number Naming for Organizing.
6) Enter Refiner Denoise (0.4 is default and good)
7) Enter Face Fix Denoise value, it should be around 0.25 - 0.45 for best results. 0.4 is default.
8) Enter Prompts for face only, for more accuracy.
9) Choose Model checkpoint you want to use.

________________________________________________________________________

Changelog v2.1

1) Added Details Enhancer in the #2 Raw and #3 Refiner workflow, you can see the following GIF to get an overview

Value Between 1 - 20 seems to give best results, you can try with higher values
Don't Put 0 in the detail enhancer, it will give black image

2) Wider CFG Support, now you can use higher CFG values and not get artifacts up to a higher level of 9 which was not possible in v2.0 version.

Backgrounds are better
Refiner like Raw Outputs in v2.1
Stable details, observe the lips, eyes and face feature in the above gif.

3) AnimateDiff Macro Motion Controller has been added, it controls the overall motion/movements of the elements of the animation, observe motion in background.

Value Up to 5 gives good results, higher than 5 can give glitchy, choppy or jittering animation.
Below 5 will have fluid, consistent motion.

4) Total Sampler Steps: Total Number of steps for ksampler. (Default value - 10)
5) Stop raw at nth step: Raw sampler will stop at this step. (Default Value - 6)
6) HighRes Fix Upscale - It upscales the Latent - (value around 1.1 - 1.4 is good, above 1.5 no change is observed)
7) Start Highres Fix from nth step: Higres fix sampler will start at this step and continue till the total sampler steps. (Default Value - 5)

____________________________________________________________________

VIDEO BREAKDOWN :

Positive Prompt : (masterpiece, top quality, best quality, official art, beautiful and aesthetic), Extreme detailed, Sexy ,1girl, blond, blue dress, blue hairband, blue eyes, blond hair

Negative Prompt : ugly, deformed, bad lighting, blurry, text, clouds, watermark, extra hands, bad quality, deformed hands, deformed fingers, nostalgic, drawing, painting, bad anatomy, worst quality, blurry, blurred, normal quality, bad focus, tripod, three legs, weird legs, short legs, bag, handbag, 3 hands, 4 hands, three hands

(embedding:BadDream:1) boy, man, male,

(embedding:ng_deepnegative_v1_75t:1),

(embedding:epiCNegative:1),

(embedding:bad-picture-chill-75v:1),

(embedding:AS-YoungV2-neg:1),

(embedding:ERA09NEGV2:1)

CFG - 4

Model - civitai.com/models/56680/imp

No Loras

ControlNet - OpenPose (Weight - 1.0) and LineArt (Weight - 0.9)

Original Video Link:
https://www.youtube.com/watch?v=AYs_ZmVvyyQ&ab_channel=KatyPerryVEVO

PROCESS :
1) Video was trimmed around the main song moment of 15 seconds.
2) It was processed in Part 1 - Line and OpenPose Passes was extracted for this Video.
3) Above mentioned prompts and settings were applied in the #2 Raw Animation workflow.
4) It was then Refined in Part 3
5) As face was looking good after refiner, #4 face fix workflow was not used.
6) Output Frames were combined, and post Processed in After Effects.

______________________________________________________________________

INSTALLATION :

All the Custom Nodes used can be downloaded easily from inside the manager and for some cases if the manager fails to install them, you can manually install with these links:

-------------------------------------------------------------------------------------------------------------------------

*Simple Evaluate Float | Integers | Strings Node error can be solved by manually installing the v1.92 from the following link and restarting Comfy as administrator to install the remaining Dependencies: Efficiency Nodes for ComfyUI - v1.92 | Stable Diffusion Other | Civitai

Or you can simply delete the A+B node and the overlapping node, the workflow will run fine without them.

-------------------------------------------------------------------------------------------------------------------------

** [Part 5] ReActor Face Swap workflow has been discarded for now as it was complicated to install and gave buggy results**

----------------------------------------

AnimateDiff Models :

1) Temporaldiff-v1-animatediff

2) MotionModel_v01

---------------------------------------------

LCM Lora : huggingface.co/latent-consistency/lcm-lora-sdv1-5/tree/main

-----------------------------------------------------------------------------------------------------

END

-----------------------------------------------------------------------------------------------------

Workflows :

Gdrive Link : ControlNet Animation Using ControlNet - Google Drive

and also attached below :

Post Update - 28/12/2023

1) The Recent Update Broke the Text Concatenate Nodes while image saving, you have to select the clean_whitespace value to true to work properly in every parts of the workflow.

2) Now you have to remove the quotes " " in the input path of the Load video Input node in Part 1 - ControlNet Passes Exporter to work properly.

The Files attached below are updated and will not give the Clean_whitespace error anymore.
.
.

My Discord Server : https://discord.gg/z9rgJyfPWJ