Control AI Actors with this Awesome Workflow (LivePortrait Tutorial)

you can now control an actor's performance using AI so if you have video footage of someone acting you can project that performance onto AI video from any tool like Runway cling or Minx plus this method Works in a variety of styles including animation the best part is you don't have to be a computer wizard to make it happen let me show you how it's done now before we get going I should note that you can download the assets that we use inside of this tutorial by clicking the link below this video all right let's get

to it so this method goes Way Beyond just normal lip syncing methods that you probably have seen online those methods are cool you can basically like type in text or upload audio and lips will match but this one actually allows you to control basically every aspect of the face using reference video so let me show you what I mean so I have this video clip here of me basically overacting I'm not good actor guys uh and I want to project this face onto other video clips from AI tools like Runway or Luma or Minimax or

any other AI video tool that you may want to use and the cool thing is you actually can so I projected that face onto this woman here you can see it did pretty decent job it also works in an animated style so we can see we have the face on this character here so let me show you the exact steps that you you need to take in order to make this happen so it all starts with a reference image you need to create an image in any AI image generator that you want to use I

went to Mid journey and typed in a woman facing a camera with a simple white background just incredibly simple but you could do a cinematic aesthetic as well with like a conversation scene it really is just up to whatever you're going for now this method works best whenever your subject doesn't have things blocking their face so glasses or hair can kind of messed things up so just keep that in mind so I went with this image of this woman here you can just download it to your machine now it's time to animate that image I'm

going to use Runway gen 3 but you could use cing or Luma laabs it's entirely up to you so I'm just going to go ahead and go to get started and we have the image of our woman here we'll drag and drop it into runway for our prompt I'm going to keep things kind of generic and just say a woman talking to the camera but I do recommend prompting in the specific emotion that you're going for in your scene if it's a sad scene you can type in sadness if it's you know excitement or joy

you can type in that and then it'll will just kind of help with the overall body language because we are going to be changing the face but you do want the uh the other you know appendages to match the feelings that you're trying to emulate so you can just type in the prompt as you see fit and we'll make sure this is in 10 seconds we're in gen 3 Alpha and you can just go ahead and generate a few times so after that generation a we have this video clip of this woman speaking here and

you can see it actually looks pretty natural but we're not able to control the face she could just be saying any gibberish because it doesn't match any sort of lip syncing or performance that we're going for and just to note it also works with animation so we have this animated shot here of this character like in Time Square and uh we'll be able to face swap over that character to control the performance so now it's time to record video footage of of your performance honestly some of those rigs like they have in Avatar where they

have a camera right in front of their face would be the best because you want the character's face to be kind of as static as possible and it can be kind of weird to act with your face completely still now I went online and looked at some of those Rigs and they're like above $200 so totally understand if you just want to record your face from a normal camera now I do recommend for the lighting to be flat so make sure that you're standing in front of a window or a softbox so that the AI

tool has just more tracking points to pull from so I have this video footage here of me talking I'm basically standing in front of a window and let's not uh critique the delivery here it's uh it's pretty terrible but you get the idea now one alternate step that you could do is motion tracking the face to stabilize it now I'm not going to get into that in this video but if you're a student at curious Refuge you'll find those Advanced lessons in our AI film making course and the last thing that we're going to need

to do is speed up the reference footage I have no idea why but it only works whenever you speed up your reference footage by 25% so I'm inside of Premiere here and I have our video clip of me our subject talking and all I'm going to do is rightclick I'm going to go to speed duration and we'll type in 125% so we'll go ahead and click enter and now we have this video footage of me talking and you can go ahead and Export that as an h264 file I'll just go ahead and hit command M

to export and we'll make sure it's h264 and you can save it to whatever folder you want okay cool so now we have this spup footage of our subject I stabilized it that's totally optional you don't have to do that now it's time to hop into the tool that we will be using for the face swapping now we're going to be using live portrait for our demo life portrait started out by allowing you to animate a face from an image using reference video but now you can do video to video now one of the problems

is life portrait uses comfy UI which is kind of scary and it can be quite technical but the good news is you don't actually have to run comfy UI on your local machine I'm working off a Mac and I can use comfy UI with an incredible computer here's how to do it basically you can go to websites like run comfy .c and run comfy UI on a cloud version of the machine it only cost a few cents to use it just for a few minutes and it's really incredible because they take care of setting everything

up for you so I'm on run comfy website and you can see they have all sorts of comfy UI models that are really interesting I'm going to go to live portrait vid to viid and go ahead and click run workflow and I'm going to select the medium mod model here you also can select the free one if you want but honestly I recommend just paying a few cents you'll use the tool for half an hour 1 hour $1 honestly small price to pay to have hyper controlled footage from AI video tools so I'm going to

go ahead and click launch and go ahead and click next now it'll take a few minutes for this machine to load up so I guess while you wait you can go over to curious refuge.com go to our free AI film making course and sign up to get a free intro to AI film making course that's completely available to you we talk about our favorite tools techniques and processes that you need to know to create an amazing AI film we train artists at all of the major Studios and we would love to have you inside of

our training okay so comy UI is loaded and this is terrifying all these nodes going everywhere the good news is there's only about three to four buttons that you actually need to know about in order to use this tool so the first one we're going to look at is this section right here with this guy in this like Park all you have to do is click choose video to upload and now what you're going to do is select the video footage the AI video plate that you want to composite your face on so I have

this video clip of this woman here and we will go ahead and click open now make sure your video clips are in that h264 format or they may not work correctly okay so we have the video footage loaded up there now it's time to load in our face animation and I loved the default Face animation here I was actually curious to project that onto some other footage that I put together earlier so I have this shot from Runway of this woman basically like just like talking in a cafe and uh and then I ran it

through the live portrait with that face and it's just Bonkers uh I want to see a film I want to see something that just has somebody with just like a crazy clown possession like that uh so now we need to swap out the base here so go ahead and click choose video to upload and you're going to upload your sped up actor's performance so there we go and we will bring it into the frame there so now our video footage is loaded up the next thing we need to do is Define the number of frames

that need to be rendered so most films are 24 frames per second so you can do the math for this one I'm just going to go in and do 200 frames for not for any specific reason we'll just do a few seconds there and we'll go ahead and zoom out you can ignore most of these settings and all you have to do is go to the end here where we have this box and we'll go ahead and click Q prompt so the entire render process will take anywhere from 30 seconds to about a minute it's

pretty quick so here is our end result here and you can see yeah we have the woman she's talking the face projection is on her face and it did a pretty good job now there is a little warping on the top and the bottom and her hair does get a little wonky in some parts so just keep that in mind whenever you're working on one of these projects now if things are a little too weird with your video what you can adjust to kind of smooth things out is the section right here it says driving

smooth observation variants not entirely sure what that exactly means but long story short if you change this number to a lower number like two what it will do is just smooth things out a little bit more and so all you have to do then is just go ahead and click Q prompt and it will render that footage Okay cool so now we have our video footage of our woman talking here and let's go ahead and right click on it and we will go to save preview now the video footage that you get from live portrait

is going to be somewhat lower in resolution so you need to upres it using an AI video upriser there are some online options like Korea AI but the tool that I'm going to be using is topaz video AI it's the tool used by professionals they use this tool for Uprising you know old archival footage and DVDs and things like that uh and it's just kind of the industry standard at this point so I'm going to take our video footage drop it into topaz video we'll select that footage and you can change the output to any

size you want we'll just go ahead and do Time 4 which would make it basically 4K and what you want to do if you go under the enhancement section go to the AI model and select Iris this one is specifically designed for Uprising people's faces so it does a good job with just kind of stabilizing the face and you can turn this recover detail down to zero that will just create more creativity in the uprising process which I appreciate and you can make this h264 if it's just going straight to the web or if you're

working on an editing project you're going to want to make sure that's in a format like prores which is more of an intermediate codec designed for working on video editing projects so when you're ready go ahead and click export and let's take a look at our result Okay cool so here is our result after using topaz video Ai and you can see it did a pretty good job now obviously it's not perfect there's still some uh artifacting there and you know parts of her hair kind of messed up but the truth is this is just

scratching the surface you now have the ability to control an actor's per performance so if you work with some incredible actors you can take their performance and put it on all sorts of different characters whether they are live action characters or animated characters now I should note that you can see in this example here we have this woman and there was kind of hair covering up her face and I think that life portrait has a hard time with connecting with track points so whenever you're in the editing process just make sure that you're selecting characters

that don't have things that are blocking their face okay thank you so much for watching this video you can go over to curious refuge.com to download the assets that were used inside of this tutorial and you can check out our courses we train artists in over 126 countries on the latest AI Trends and techniques and of course you can subscribe here on YouTube to get the latest tutorials directly on the platform if you have any suggestions for future tutorials be sure to leave a comment below and request what you want to see in the next

video thank you so much for watching we'll see you next time