Train Your Own Flux.1 LoRA

I'd start with getting to grips with workflow basics - https://www.youtube.com/watch?v=VM9snsuoqBc - after that, the world is your oyster!

Nerdy Rodent

Hi im having this custom ui and can you help how can i integrate some of the nodes you use like folrence2Run for the text output and kj nodes for images size same as my input can you help or connect nodes and give me jason file using my jason file i want have to them in my ui https://drive.google.com/file/d/1hnVo4ClahJHCV5XP14ZEfDpz7ZnZ8vBI/view?usp=sharing

Rart

I have the same issue. Have you solved it?

Robert-Marcus Koch

Make sure to check you've got everything from the links provided in the workflow as it's likely you're missing the last one in the list: https://github.com/ltdrdata/ComfyUI-Inspire-Pack https://github.com/kijai/ComfyUI-Florence2 https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes https://github.com/WASasquatch/was-node-suite-comfyui https://github.com/ALatentPlace/ComfyUI_yanc

Nerdy Rodent

same issue

jake

When I load the Florence_Captioner, the Save Image node is still red even after installing all missing nodes. Any ideas?

ed.enai

yay! :)

Nerdy Rodent

it`s working I did it ! (igore previous comment)

Peter Rossa

thx, I think I will do it all again xD I got another error: D:\FreshSDXL\AI_toolkit__OLD\ai-toolkit>python run.py config/train_lora_flux_24gb_example2.yaml Traceback (most recent call last): File "D:\FreshSDXL\AI_toolkit__OLD\ai-toolkit\run.py", line 22, in from toolkit.job import get_job File "D:\FreshSDXL\AI_toolkit__OLD\ai-toolkit\toolkit\job.py", line 3, in from toolkit.config import get_config File "D:\FreshSDXL\AI_toolkit__OLD\ai-toolkit\toolkit\config.py", line 5, in import oyaml as yaml ModuleNotFoundError: No module named 'oyaml'

Peter Rossa

You save your .env file in the root directory and in "HF_TOKEN=your_key_here" replace 'your_key_here' with your actual key (from after you login to huggingface). The text file called .env is unrelated to the python module. From your example output there, it looks like you've forgotten to activate your ai-toolkit environment first - hence it can't find anything

Nerdy Rodent

sorry for spamming xD ok I have failed on creating .env file, Im doing something wrong I dont know where to put the .env file and how to exacly activate my READ token, after executing training I got this messege: D:\FreshSDXL\AI_toolkit\ai-toolkit>python run.py config/train_lora_flux_24gb_example2.yaml Traceback (most recent call last): File "D:\FreshSDXL\AI_toolkit\ai-toolkit\run.py", line 5, in from dotenv import load_dotenv ModuleNotFoundError: No module named 'dotenv'

Peter Rossa

npmjs dot com / package/dotenv#-install

NetDude

Peter Rossa

(nevermind on that custom save image node I had to reload the worklflow)

Peter Rossa

so I m learning to do that training, I was updating nodes via manager and now ComfyUI doesnt work on Opera (it just opens a blank white tab with no nodes nothing) but it does work in firefox, weird xD, (and I cant get that fancy save image custom node to work I did cloned it from here: https://github.com/ALatentPlace/ComfyUI_yanc) but it still shows up red in the captioning workflow

Peter Rossa

Start with a clean install of Comfy so you know you've got a working install, then add the extra nodes from there :)

Nerdy Rodent

No matter what I tried, I just couldn't get this to work.

ThoughtFission

maybe your images start at 1 but the text at 0, you should make sure it both starts at the same number

Ayatanne

The Flux Dev license doesn't allow me to put the files on Patreon, commercial use and all that, but I got the dataset images from here - https://www.metmuseum.org/art/collection/search?q=ukiyo-e&showOnly=openAccess :)

Nerdy Rodent

The notes in the workflow have the links you'll need, which are: https://github.com/ltdrdata/ComfyUI-Inspire-Pack https://github.com/kijai/ComfyUI-Florence2 https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes https://github.com/WASasquatch/was-node-suite-comfyui https://github.com/ALatentPlace/ComfyUI_yanc

Nerdy Rodent

It's likely you've got conflicting nodes installed, so it's using the other one you have instead of the actual one in the workflow. Try uninstalling or disabling any conflicting nodes.

Nerdy Rodent

:)

Nerdy Rodent

The captions don't correspond to all images. The number of txt is not associated to the good image :(

Guillaume C.

I had the same issue, but no solution for the moment. I deleted the nodes, it saved some txt files, but the result seems good only for 3-4 files, after the caption doesn't correspond to the image

Guillaume C.

Can you share your Woodblock art style lora, I like it a lot

Ayatanne

I have the same problem and I installed that from your link, but the save image problem is still there.

Ayatanne

I have the same problem and install missing is empty.

Ayatanne

When loading the captioner, it says Warning: Missing Node Types When loading the graph, the following node types were not found: > Save Image No selected item Nodes that have failed to load will show as red on the graph. But when I want to install the missing ones with the manager it cant find anything. I used your git link from below but it is still not working.

Ayatanne

This was an incredible video + guide. This was my first dip into the AI pool and everything worked like a charm. I created a Patreon account specifically to follow your page. Thanks for the incredible write up.

omgwtfh4x

Excellent! ComfyUI Manager is how I install my nodes and it's not let me down yet! For your error, does the specified path exist? Likely it's a pinokio specific issue, so it may be worth going with a standard install instead to test.

Nerdy Rodent

I already have the manager. I updated again, and it seems better now. But I always have an issue with the "save_image". (stay red) And in the terminal I ve this error message To see the GUI go to: http://127.0.0.1:8188 [ERROR] An error occurred while retrieving information for the 'Florence2ModelLoader' node. Traceback (most recent call last): File "m:\pinokio\api\comfy.git\app\server.py", line 468, in get_object_info out[x] = node_info(x) File "m:\pinokio\api\comfy.git\app\server.py", line 436, in node_info info['input'] = obj_class.INPUT_TYPES() File "M:\pinokio\api\comfy.git\app\custom_nodes\ComfyUI-Florence2\nodes.py", line 142, in INPUT_TYPES "model": ([item.name for item in Path(folder_paths.models_dir, "LLM").iterdir() if item.is_dir()],) , File "M:\pinokio\api\comfy.git\app\custom_nodes\ComfyUI-Florence2\nodes.py", line 142, in "model": ([item.name for item in Path(folder_paths.models_dir, "LLM").iterdir() if item.is_dir()],) , File "m:\pinokio\bin\miniconda\lib\pathlib.py", line 1017, in iterdir for name in self._accessor.listdir(self): FileNotFoundError: [WinError 3] Le chemin d’accès spécifié est introuvable: 'M:\\pinokio\\api\\comfy.gi t\\app\\models\\LLM' [Inspire Pack] IPAdapterPlus is not installed. Error. No naistyles.csv found. Put your naistyles.csv in the custom_nodes/ComfyUI_NAI-mod/CSV directory of ComfyUI. Then press "Refresh". Your current root directory is: M:\pinokio\api\comfy.git\app Error. No naistyles.csv found. Put your naistyles.csv in the custom_nodes/ComfyUI_NAI-mod/CSV directory of ComfyUI. Then press "Refresh". Your current root directory is: M:\pinokio\api\comfy.git\app

Guillaume C.

As a first-time user of ComfyUI I highly suggest installing ComfyUI manager first - https://github.com/ltdrdata/ComfyUI-Manager - which can then be used to install any missing nodes :)

Nerdy Rodent

Hello When I launch the florence project, I have a couple of issues with unknown nodes Warning: Missing Node TypesWhen loading the graph, the following node types were not found: DownloadAndLoad Florence2ModelFlorenc2Run> Save ImageListCounter //Inspire LoadImageListFromDir //Inspire

Guillaume C.

A few workarounds off the top of my head: 1. If you're using Microsoft Windows, instead boot a Linux Live CD to do the download 2. Use git clone to download the repo instead of using a web browser 3. Use huggingface-cli to download the repo instead of using a web browser 4. Use a public internet service (such as a local library, or internet cafe) and transfer the files back to your system on a USB drive

Nerdy Rodent

Sorry to be a pain in the ass. I tried several proxy servers and VPN but same result. I am trying almost a week now. It's frustrating but I am still not willing to give up. Is there maybe a workaround to solve this problem?

Jens Huebner

Worked like a charm, little gnawing creature! LORA training is in progress. Thanks a bunch!

Lúcio Nunes

Try again after updating - https://github.com/ostris/ai-toolkit/issues/123

Nerdy Rodent

Good day, Nerdy and folks. I'm getting this error when trying to train my lora: python run.py train_lora_flux_24gb_luciov0.yaml Running 1 job Error running job: cannot import name 'apply_rope' from 'diffusers.models.attention_processor' (/home/comfyui/ai-toolkit/ai-toolkit-env/lib/python3.12/site-packages/diffusers/models/attention_processor.py) Any ideas how to get this resolved?

Lúcio Nunes

Assuming you're using the unzipped ComfyUI portable download (https://github.com/comfyanonymous/ComfyUI/releases/tag/v0.0.8), you'll need to use "run_nvidia_gpu.bat" to start it. As a first time Comfy user, you'll also want to install ComfyUI manager using "install-manager-for-portable-version.bat" - https://github.com/ltdrdata/ComfyUI-Manager#installationmethod2-installation-for-portable-comfyui-version-comfyui-manager-only

Nerdy Rodent

I don't see tensorops listed in the requirements file for either Florence2 or LLM Party, so not sure what's going on there. Check for errors as ComfyUI starts up, or as an alternative you could start up a new, clean comfyui environment which should be free of any issues.

Nerdy Rodent

Their GitHub issues tab is available at https://github.com/kijai/ComfyUI-Florence2/issues?q=is%3Aissue+ - I did a search for "mps" but there weren't any results, so you may need to create a new one.

Nerdy Rodent

Ok. I just joined. I did the install, apparently successfully. I wanted to run the workflow to create the training data. But, lo and behold, no start.bat for comfyui. How do i initiate comfyui. I have a few other installations in other directories. But, is there one in the ai-toolkit directory that this installation created?

Gary

after I use install nodes, tensorops fails to install, either from manager or comman line, even with a 'pip install requirments"

Albert Bukoski

yes you are right, I'm on mac, but I don't understand what you try explain me, where do I need to go exactly?

Randall Barriga

It could be something to do with the country you're in, so perhaps a VPN would aid the internet connection? Essentially, you need some way to download the files from huggingface - either by letting it do it automatically, via a web browser, or any other downloader

Nerdy Rodent

Sounds like you're probably on an Apple Mac? One for their github issues by the sounds of things!

Nerdy Rodent

Thanks again. That worked. But now I got stucked while downloading shards. It just isn't do anything. I waited several hours yesterday but it still was 0%. Any ideas what I can do? Running 1 process Loading Flux model Loading transformer Quantizing transformer Loading vae Loading t5 You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers Downloading shards: 0%| 0/2 [00:00?, ?it/s] model-00001-of-00002.safetensors: 0%| 0.00/4.99G [00:00?, ?B/s]

Jens Huebner

Hello I have error when executing Florence2Run, The operator 'aten::isin.Tensor_Tensor_out' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS. do you know how to fix that?

Randall Barriga

Their page says, "12/15/2023 WAS-NS is not under active development." - it's okay, I'll just live with it. Thanks.

HealingPaint

Thanks, that's probably the easiest solution and I'll try that.

Runebinder

Strange, as I can set it to 0 without issue. Could be a Microsoft Windows thing. Maybe worth raising an issue at https://github.com/WASasquatch/was-node-suite-comfyui

Nerdy Rodent

You can also download the files manually from https://huggingface.co/black-forest-labs/FLUX.1-dev They go into your default huggingface cache directory - https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache - e.g ~/.cache/huggingface/hub

Nerdy Rodent

At least I can train now when I rename the TEXT files to match, but I don't know why the workflow is forcing me to use a filename number padding of 2 or higher...can that be fixed?

HealingPaint

Changing it to 2 fixed it, but it renames the files not to match the images which are 1.png, 2.png, etc...not the text files are 001.txt, 101.txt, etc ?

HealingPaint

I do see this in the command window as well and the text module in comfy is bordered in red. Failed to validate prompt for output 259: * (prompt): - Value 0 smaller than min of 2: filename_number_padding * Save Text File 259: - Value 0 smaller than min of 2: filename_number_padding Output will be ignored

HealingPaint

Hi, set everything up (on Windows 11), run the command to start training the LoRA and get this error: "OSError: black-forest-labs/FLUX.1-dev does not appear to have a file named config.json". I've deleted the .yaml file and recopied from examples folder and only changed the path to where the sample set it in case I messed something up the first time but same error. I've created the .env file with my Hugging Face token but not joy. Not very good with Python, any ideas?

Runebinder

The paths are both copy-pasted, identical. tried again, careful with spaces, etc...same issue. No idea. I need to find a different way to caption these images somehow.

HealingPaint

Very strange! As you're getting the images output to the directory you specify it won't be a permissions issue. Everything still points to a typo in your text output path. Maybe try making the directory first, then copy & paste the path in?

Nerdy Rodent

Sadly no text files are found. It does show the images (in the workflow itself) after running it. It shows TEXT in CMD and in the SHOW TEXT module in the workflow itself which reveals that the CLIP is working to describe the images, but no text files are being created... it sucks to be this close and not able to do this. :-( (I'm just using the workflow defaults and changing the path only BTW)

HealingPaint

It'll be looking in your default huggingface cache directory - https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache - e.g ~/.cache/huggingface/hub

Nerdy Rodent

Thank you, that did the trick. But now I got stuck with downloading the diffusion_pytorch_model-00001-of-00003.safetensors ... it just didn't do anything, even after waiting one hour it was still 0% So I am loading them manually now from https://huggingface.co/black-forest-labs/FLUX.1-dev/tree/main/transformer I think I have to put all five files from there into this folder? E:\ai-toolkit\checkpoints\flux-dev\transformer Can someome confirm this? Sorry if this is a stupid question. I am totally new to all this.

Jens Huebner

Try searching for the file 0.txt in case you've got a typo in your directory path and it's not saving where you think

Nerdy Rodent

I have all 3 paths set. the images complete fine but the TXT files never appear in any path I set. It acts like it will work, shows the descriptions in the CMD window, but they don't appear on disk. Ideas? I'm setting the same path for both images and txt files.

HealingPaint

Have you installed the one from here - https://github.com/ALatentPlace/ComfyUI_yanc

Nerdy Rodent

I am having the same issue. Any idea on how to fix that?

Saiphan Brunner

Did you run "git submodule update --init --recursive"?

Nerdy Rodent

Thanks for the tip, I was struggling with this also.

Alexander

After creating a image folder, when I try to run it, i get: Running 1 job Error running job: No module named 'ipadapter.ip_adapter' Any Ideas how I can fix that?

Jens Huebner

you'll need to set 3 paths: 1. The directory with all your images 2. The full path to where you want to save the text files (e.g. /home/myuser/comfy/output/my-lora) 3. The path relative to your ComfyUI output directory to save the image files (e.g. my-lora)

Nerdy Rodent

Nope. It does all that for you :)

Nerdy Rodent

Now it works to output images, and shows the TEXT being figured out in the CMD window, but no TXT files are appearing in the folder...I've tried a few output folders for the text as well...ideas?

HealingPaint

does resolution/aspect ratios of image data set matter at all?? from your video it looks like not. it’s usually the most tedious part of training for me. o happy day if i don’t have to 512x512 my lora fodder anymore

Ronny Khalil

Yup that works, kept reloading it and no more errors loading up! Thanks.

HealingPaint

I had exactly the same issue, even after installing YANC in custom nodes, and even though the included save images node was missing inputs. However, after reloading workflow, it finally did work for me. How? No idea.

Jeroen Van Harten

Is there another way to get my images captioned quickly? I'm stuck here before I can run the training I have otherwise ready to go...

HealingPaint

There are no conflicting nodes, so install missing should work... but the one for saving is from here - https://github.com/ALatentPlace/ComfyUI_yanc

Nerdy Rodent

yes yes yessss

Ronny Khalil

I've installed all missing modules in Comfy, and added some SAVE IMAGE ones, but cannot clear this error..."When loading the graph, the following node types were not found: Save Image" what am I missing? What is the exact addon I need to install for this to load the Florence Captioner workflow?

HealingPaint