FLF with Videos?
Hey, I recently found this workflow
https://huggingface.co/siraxe/MergeGreen_IC-lora_ltx2.3
The idea is that you upload two videos, you can extra from 18 to 24 frames out of end of first clip and beginning of second clip, then this wf with lora generates a bridge clip between those clips with the length you prefer (for example 121 frames), it's like FLF workflow but model has much more context (obviously, it's not just 1 frame) and no brightness issue, flickers like last frame often does in LTX. So output is 121 frames, 18 or 24 (you decide) frames in the beginning is end of your first clip and last 18-24 frames are beginning of second clip, perfect transition, solves the issue with last frame in LTX, quality is also nice.
But here's two major issues with this wf and lora. Color mismatch, since it generates in RGB field clips are often more purple or if you set too high dimensions or too long video then you get a huge green tint (i assume lora wasn't trained on long clips and high resolution). Also major issue is sound it produces, since it produces it out of green screen you often get just noise, music or whatever, completely off.
So maybe you can take this as an inspiration for a workflow in the future like FLF but with videos, it solves end frame issue so I think will be very helpful for a lot of people, if you manage how to force LTX to extract frames from video automatically and use them to generate a clip or how to improve this workflow so it doesn't ruin colors match and no sound issues that would be amazing. I spent my day trying both of these ideas - unsuccessful. You're the last hope here
Yes I was also curious about this workflow / lora.
I used the "exact" way its made in the repro, but recrated it with basic standard comfyUI nodes to simplify (front end view, its complicated behind the nodes..hehe)
A test run here: https://huggingface.co/Kijai/LTX2.3_comfy/discussions/32#69fdee0a750941a68acdbe05
I didnt upload it yet, wanted to tweak it a bit, and test it a bit more, to see if its all working well.
But thats totally based on the green lora way. It could be made completely differently using LTX "Re-Take" feature, where an arbitrary length gap between 2 videos are just masked with standard masking.
And then feed LTX the last reference frames (and audio) from the first video as start before the mask, and the first frames of next video (and audio), as end reference frames.
And let LTX do its magic. If it works better or worse, i dont really know. Its just been on my list of things I want to try ;-)
Will see if i can give it a try soon, and the Green Lora version that is a bit simpler (hopefully) I'll upload asap.
Of course this was a bit of a "hopeless" test run of my own version of the green screen lora wf. Since the 2 videos are way different, the green screen lora had its challenges i bet, maybe not a fair test run ;-)
And didn't put a lot of thought into prompting the transition, so I bet the result could be a lot better with some tweaks and thoughts..
But you get the idea.. and yes, I noticed the same, that the audio would often distort a bit, and be a bit weird.
Will see if i can remedy that a bit. Maybe leaving a bigger gap for LTX to figure things out between audio A and audio B
And interestingly a new lora just released that is aiming at somewhat the same
https://huggingface.co/systms/SYSTMS-FLW-IC-LORA-LTX-2.3
will make a workflow for both green screen and this new lora ;-)
same video example, but with Systms FLW lora
They work pretty much the same way ish .. this lora uses gray screen instead of green screen
hell yes. I also tested a gray screen one, but for whatever reason it keeps making me gray flickers. Managed to move them at the very beginning of the video so i can at least trim them, but comparing both of them, green one is definitely winning for my usecase, speed of generation was also a bit faster
i did generate around 50-100 clips with the green lora one already adjusting all the little settings i could recognize to keep that quality/generationtime balance and for my use case (collecting all those tiny clips from i2v and transition wfs and making one long video in davinci to keep a very good quality and consistency across the whole video) there are some tiny things such as slight color and brightness mismatch from the original clip (using colormatch node helps a bit and postproduction), lack of tiny details (when eyes move especially), can't run 1536p (green tint all over the screen, 1080p is fine) and lack of audio (gray one added custom audio node to cover it). I still do work with this lora and happy with the results and a huge respect to the guy who made it, it gives a room for inspiration since quality degradation on longer clips is a bottleneck
Yes tried it a few times as well, its a nice little lora (both of them) for a transition effect.
The audio part I am trying to see if i can make it transition as well, instead of the custom audio... but so far its a bit wonky, LTX doesnt seem to accept the audio mask as partial ... ;-) but will try a few other things
I'd love to try this @RuneXX when its ready
I've been looking for a good way to make very short cut transitions (0.5-2 seconds) between two videos for a while
e.g joining videos of a person talking into one long video but transitions in between which morph the body movements together more naturally rather that cut
Do you think it could do that?
e.g joining videos of a person talking into one long video but transitions in between which morph the body movements together more naturally rather that cut
Do you think it could do that?
Should probably work well for that yes ;-)
First one, the Systms FLW lora https://huggingface.co/RuneXX/LTX-2.3-Workflows/discussions/147
Uploading the Merge green one too asap.
Both workflows can be used interchangably even if they are tiny bit different (one is IC-Guide Lora Guide, other is regular LTX Guider, to be in line with the lora makers own suggestions).
Even if so, you can perfectly fine use the Merge Green lora in the Systms FLW workflow, and vice versa.
All you need to do is change the mask color value. Its an easy input for that in the workflow
Systms FLW uses gray mask, and Merge Green uses green mask
Also made a more generic one that simply uses LTX inpainting ("Re-Take")
That can be used for all sorts of things.. . transition between clips, extending a scene between clips, etc etc
It does not depend on any lora
https://huggingface.co/RuneXX/LTX-2.3-Workflows/discussions/149