AN INTRODUCTION TO VIDEO CAMERA MATCHING WITH RS3D

    Another tutorial in the "Saved From The Abyss" Series , which are tutorials concerning little known but very useful information that may have very briefly appeared on the Realsoft3D Mailing List .

Introduction

    There is a ton of entertainment shown today on television and of course, cinema , that use a blend of live action and computer generated content . The art form is so seamless in it's presentation it is almost impossible to determine what is which (well , the 2 headed aliens are more than likely CGI) or exactly what part of the room or outdoor shot is real , and how much of it is done with 3D computer S/W ?

    RS3D provides many elaborate tools designed to easily render synthetic scene objects at a level of photorealistic quality far beyond what is required for typical video insertion . The 3D Software industry seems to be devoted to realism , yet , in fact , one of the easiest way's to "sell" your highly realistic renders is to simly place them into a video or photograph , in such a manner that the CGI objects are actually enhanced by their realworld surroundings ! Really !

    Anyway , hopefully this tutorial will let you in on how to begin to experiment with combining video and your very own RS3D CGI , using freeware S/W's , like Icarus or Voodoo and Virtualdub . Realsoft has provided several import plugins for importing realworld camera data (via camera tracking software) , and automatically creating an exact synthetic version of any camera inside RS3D .

    This tutorial will not deal with 'Image Aspect Ratio' or the RS3D camera . I highly recommend you visit Arjo's excellent primer  for info on that topic after you've done this tutorial , as a means of better understanding some very important issues you will sooner or later have to come to terms with .

    The skills involved with matching shadows or lighting will be left to another time , perhaps . It is already quite a lengthy "One-Page Tutorial" and really is only about getting data into Realsoft3D , and how to match a RS3D object with that imported data .

    Once again I should also mention that it written for the beginner , thus the rather long-winded manner . Apologies to the advanced RS3D users who must muddle through the basics in their effort to learn something new .

A Simple Realsoft3D Example

    Here's what we are aiming for in this very basic exercise ... our resulting animation  (Xvid 400KB)   **NOTE** - Here's another  recent example  (Xvid 4MB)

That is , to blend a RS3D 3D Object with a 2D Object from the Voodoo camera matching S/W's included Free-Move example . Select a slow play speed in MS Media Player and load or view it in     XnView    or   Virtualdub  in a frame by frame manner , or at whatever speed you desire via the play controls .

    The above animation came from the RS3D project file that is   included in this .zip file  Grab the .zip file and load the project into RS3D . The .jpeg's are included , and all that may be required is to direct the paths to the project's materials . Load it and do that now , have a look around inside the project , then save and close the project and start the tutorial , which will begin with focusing on the actual camera tracking Software ... Icarus .

THE SOFTWARE

     Icarus has been defunct for years now , since it's technology was bought out , reworked and is now being sold as a    Pixel Farm Product   . Icarus can still be used for non-commercial and educational uses , but not for profit . Google still shows a couple of places on the net that have it available for download .

    In all fairness to you and to the S/W developers , I should mention that RS3D does import several other commercial camera tracking packages , and since I haven't really used any , I can't vouch for them here . I have seen the SynthEyes demo   SynthEyes's Web Site  . and found the program to at least look very comprehensive and appears to be a bargain at $399 USD . Excellent tech support and an active forum as well .

     SynthEyes does have a very good example of a camera flyover of a pyramid object , with an output script in a format that is written specifically for RS3D . It is java based and looks to be an excellent "one-step handshake" between it and RS3D . The SynthEyes Method seems to provide a truly professional solution . Download the image sequence & camera output script from the SynthEyes site , and then import both into RS3D and judge for yourself .

This tutorial and Realsoft3D project is based on another freeware and open sourced camera tracking software - Voodoo . RS3D does not import Voodoo output but we will use one of the Voodoo S/W installs image sequences , (a large yellow tractor) since it is a very good example of large camera movements around a well defined subject .

LET'S TRY IT !

    OK , now that the introduction is over with , let's get quickly into this very simple process of creating camera data and then importing it into RS3D . We'll go through these steps in a general way that may apply for any video or image sequence , not just the included .jpeg image sequence . Do attempt to match my results using Icarus by creating your own Realsoft3D camera and see how it compares to the Icarus export in the above-mentioned .zip file .

     STEP #1

Icarus is a little picky about what kinds of formats it can import . If the video or animation that you want to use is in an image format , such as .jpeg or .bmp , or if it's an mpeg , open   Virtualdub  and load that image sequence and export it as an uncompressed .AVI  (for Icarus in the next step , and for RS3D , later) . Uncompressed is always best ... no compression artifacts to confuse the camera tracking S/W .

     STEP #2

Open Icarus by double clicking on Calibration.exe in the "Icarus/bin" folder. Create a new Project (Project/NEW) and import the .AVI (Project/Import Movie) . The Camera Parameters Control Panel will open automatically . Icarus default is 'Free Motion' and 'Constant Focal' . Change 'Constant Focal' to 'Fixed Focal' (using the pulldown menu) and then choose your type of camera from the pulldown list (see image below) .

Then , type in your camera's focal length . Leave everything else as is and choose 'Close' . If you have a video with unknown source then just leave these parameters as is for now . If the camera that Icarus builds for you does not seem to be working in Realsoft3D (or if the exported 'Features' do not match as well as Icarus indicates) you may have to come back to this Camera Parameters control and try again . Choose 'Constant Focal' instead of 'Fixed' and let Icarus try again for you . You may also have to experiment with various CCD sizes if you are working with a difficult project .

If Icarus is having trouble with a video clip , try running the whole video segment through , not just an edited clip . The more frames the better . As a last resort for problems with determining proper Focal Length , try running the video through the freeware "Voodoo" camera tracking software . It has more modern algorythms and seems to solve easier , as does Syntheyes .    Voodoo Web Site  

I should mention that for best results we should always use a video with a constant focal length (no zooming) . If you have a video that contains zooms you are free to attempt a matching by choosing the 'Variable Focal' control (from the pulldown menu) but I can not recommend it simply because I've never attempted it .

     STEP #3

Under the Tracking Menu , select 'Set Start Frame , then move the slider , to the last frame of the sequence and then 'Set End Frame' . Move back to the first frame and then under the Camera Menu , Select 'Track and Calibrate' . Icarus will look at possible points (Features) in your video to track using the default settings . Allow Icarus some time to do this .

     STEP #4

OK , it's done tracking possible feature points so we can have a quick look at the data points that have are showing in the View window . Notice how Icarus has placed bright green points on prevalent positions . You should see the exact same thing inside RS3D . Let's get a feel for what Icarus has done ... Select 'View Window' from the 'View' pulldown menu and a new window will open showing you the Icarus camera and the 'Feature' points that have been determined .

Pan and move a zoom around in this view and see if any of the points seem to portray a pattern that may loosly resemble objects in our video . Hopefully we will be able to pick out an object or 2 , but if not , don't worry , it's not critical .

One more quick look at some other data and we are ready to export the camera . At the far left on the vertical panel are the Project's Parameters . Select the Camera Parameter object and expand it . Then select and expand the other parameter boxes . Select any of the Camera 'Translation' or 'Rotation' elements and you'll notice a graph will appear at the very bottom .

This wavy line represents our camera's path through 3 dimensional space , as well as the camera's movement on it's own internal axis . Now , before we leave Icarus we want to check for 3 things . Select the 'Focal Length' element from this parameter list and we should see a straight line in the Graphing Window at bottom . Remember , we selected 'Fixed Focal' so this line must be straight , which tells us that the camera's focal length did not change during the course of the video .

Do the same for 'Aspect Ratio' and 'Skew' . These should also be straight-line (constant) data . Same for our Principal Points on the X and Y axis . If these particular parameters are not straight , then we will have to re-do the camera tracking and calibrating . Checking for this now will prevent a lot of possible time-wasting inside Realsoft3D while trying to figure out why things are not matching properly .

     STEP #5

We can finally export the tracking data and those feature points in a file that can be imported into RS3D . Select , from the 'Project' menu , 'Export 3D Motion' , and then when the file saving control opens , under 'File Type' scroll down to 'Human Readable (.txt)' . Make sure the Auto Features is set to 'ALL' , then choose a directory and a file name and click on 'OK' . This will export the 3D position of the camera , as well as the 3D position of the scene's prominent 'Features' . Close Icarus .

THAT PART IS ALL DONE .

    Now we can start RS3D and simply load the human readable text file . I prefer to work from a top view , so choose the top view and make sure you are using an 'Orthographic View' (not a 'Perspective' one) . Use the general 'File/Load' command , or go down to 'File/Import/Icarus' instead . An Icarus loader control will pop open and present you with an option to size the scene . I also prefer to load at "Scale" 0.10 , but for now just use Scale 1.0 and unselect 'Replace' .

     Don't look for the Icarus data points until you have done 2 things . First , right click in the view window and go down to "Camera/Tracking" and make sure it is unselected . This way we can move freely and not have to worry about accidentally being forced to look through the lens of the 'Icarus Camera Object' , at least not until we have a good look at this scene from a top view .

    Second step is to hit the rewind button on your 'Animation Control Bar' . If you don't already have one open , open one now and hit rewind and make sure we are at the very first frame (Frame #0 by default) . You probably won't see much yet , so select the Icarus "Features" object in the "Geometric Objects Tab" (first Tab in the Select Window) and "Drag&Drop" it into your viewport . Hit the space bar to enter edit mode . We won't edit them but your Feature Points will now have their "Red" edit boxes visible , making them much easier to see .

     Now , zoom way out (or drag & drop the project's 'Root' into the viewport window) and then select the "Icarus" object in the "Geometric Objects Window" and observe the viewport window . A group of handles just jumped to the location of this , the Icarus Camera Object . Zoom in until you can easily make out what the Icarus Camera Object looks like from above , and where it is pointing to (the features , of course) .

LET'S GET TO WORK .

     We may be getting the feeling that the scene is a fairly large one . Actually , this particular scene is not too bad , as we can see if we "Drag&Drop" the "Root" object into the 'Viewport' , but some scenes can be huge, depending on how the features have been laid out by Icarus . Some feature points could be hundreds of meters away . We can delete them later , or we could have chosen to just export certain features from Icarus . You'll get the hang of that later , but for now let's just try and get a "feel" for the size of this test scene .

    If you are zoomed into the camera in a top view , zoom out until you can see the camera and also a fair number of Features . Now , create a Camera (Tools/Creation/Create Camera) for this view . This will be our working view (or one of them) , so rename this camera as (Work-Top) and save the File as (File/Save As ) " Icarus-test-1 " . Feel free to create other camera's , and even some objects that will help you see where certain features are located . For instance , we could place a large rectangle under the main group of Feature Points .

    If we look at the layout of features (from above) we may be able to distinguish a shape , roughly where the large Tractor was sitting in that sequence of images that Icarus tracked . For now , let's place an Analytic Cube there amongst that collection of Feature Points that seem to form a square-ish looking angle (that is actually representing the tractor ) . Create a cube , rotate it , and scale it so that it roughly fills those FP's (Feature Points) then take a side view . You will probably have to extend the Cube's Depth . Do so in this side view via the Cube's Property Window , or with the Cube's Handles .

    If you're fairly happy with the Cube's size and placement (remember , this is just a test) then "Drag&Drop" the Icarus Camera Object into the View Window . There ... we are now looking through a CGI camera that Voodoo and Icarus decided must have been used to film those tractor images . How does your new Cube look ? Of course it's impossible to tell , without those tractor images there to compare it to .

Let's Add The Background Video

    Yes , we are certainly missing one key ingredient here ... the .TGA image sequence . We will apply that image sequence as a Camera Backdrop , so select the Icarus Camera and then from the Menu , choose "Tools/Creation/Create Backdrop Image" . In the requester that appears , in the 'New Image' field - select the .TGA image sequence we loaded into Voodoo , or the uncompressed .AVI that we loaded into Icarus . Set it's distance to 450 , so that it will be visible , but not in the way or blocking the view of our new Cube . Click "Accept"

    RS3D will have automatically created a new shader for you that will apply the .TGA's or the .AVI to the new 'Backdrop' for you , as the animation plays or as you scrub the timeline . To be assured this is going to happen smoothly for you , click on the second Tab in the Select Window , the Material Tab , and salect our new 'BackDrop' shader (the very last shader at the bottom of the Tab) .

    Double click on our new 'BackDrop' shader , then Click on the 'Advanced' Box & on 'Preview' (both at the Top Left of the Shader Control Panel) . An image should appear at Top Right , and a few lines of code in the 'Shader Control' area , below . The line of code 'illumination=Texture(mapcoords)' represents our backdrop images . Select that line and then click on the 'Animation Tab' and change the Animation Type from 'None' to 'Automatic' . RS3D will instantly change the necessary corresponding fields , also automatically for you . Now that's all ready to go , so go back to the Geometric Objects Tab of the Select Window .

Using OPENGL For Our Backdrop

    First make sure that you are using an OpenGL canvas and not just a wireframe view . Right Click in the View Window and at the very bottom , select 'View Properties' . On the View Properties Control Window that appears , select the 'Draw Tab' and in the 'Canvas Options' section , select 'OpenGL Shaded' .

You had previously Dragged the Icarus Camera Object into the View window , so now you should see the image (or first frame of the .AVI) of the tractor in the Viewport Window . Drag that camera in again if required . The image is a very blurry OpenGL rendering , so let's fix that now .

    Double click on the Backdrop Object (that's in the Icarus Camera Level Object) and select the "SPEC" (Special) Tab in it's Property Window. We can see that it is set for 450.000 , and if we need to adjust that number later , this is the place to do so . Now , select the Wire Tab and change the Texture Quality from the default "64" to "512" . Wow , big difference !

Congratulations ! You're Almost There

    Well , OK , how does your New Cube look now ? Adjust it with the Numeric Inputs that are on the 'Spec Tab' of it's Property Window , until it looks ... not bad (or so) . Now , let's take that quantum leap ! In the View Window , right click and go back to "Camera/Tracking" , and this time make sure it is selected. Now , scrub through your 21 frame Timeline by grabbing the Timeline Slider .

    Does your cube kind of seem to move in a manner that is relative to how the tractor seems to move ? Probably not , at least not on your very first try , so Drag & Drop your Top View Camera Object into the View window and zoom in on your cube (switch back to wireframe mode for best visibility) . Zoom right in and place your cube so that it lines up better , with the rough approximation of the Feature Points representation of the back of the Tractor .

    Take a side view and tweak the cube some more . Repeat these steps until you think you can see the cube is moving in a similiar manner as the tractor is in the images . As you may have guessed , your cube is not actually moving at all , but only appears to be doing so because of the Icarus Camera's movement . The tractor also is not moving , (nor the Backdrop) but only appears to be .

A Final Test of Proof

    Let's leave the cube (make it wireframe invisible or delete it) , and as a final test , let's try and replace the tractor's exaust pipe with a RS3D synthetic version . Zoom into the FP's from your overhead camera's vantage point , until you begin to see something like the image above . I've placed a long cylinder at this group of 6 FP's (feature points) which Icarus has captured in it's tracking data . We should note that Icarus just happened to capture this exhaust pipe as a series of feature points . However , I could have manually selected these points , or even created these points inside Icarus , for export , to be included in the Human Readable text file (see the Icarus documentation) .

    Do the same and create a cylinder , then take a side view and tweak the cylinder some more , stretching it out and moving it up or down until it seems to sit amongst the FP's , where it should . Drop the Icarus camera into the Viewport and then move , size and rotate the cylinder (only by it's handles) , until it is exactly covering up the tractors exaust pipe . Now scrub through the timeline again , slowly and frame-by-frame , carefully watching to see if the exhaust pipe is covered properly in every frame .

    If your cylinder is "drifting" , sometimes to the right of the exhaust pipe , and then sometimes to the left , you probably do not have the cylinder at the proper Z-depth (distance from the camera) , but that is easy to fix . Just take your overhead camera view of this area and move the cylinder back (away from the camera) , then scrub the timeline again . If it is better now , move it back more , if it is worse , then just move it forward (toward the camera) and try again . When you think you're close do an anim preview , or render out . The 'File/Render Window' is set to go , but you will have to choose a file directory for the output animation .

Just Practice

    That's about it . It is actually a matter of trial & error ... practice , really . In order for a person to get a good feel for , what has often been called "The Black Art of Camera Matching" , one must just keep practising , it's that simple . With Icarus , it's important to either have a lot of frames or even better , frames with a lot of movement to help determine what the camera's parameters are .

     Truthfully , this is just one step along the way . Once you do finally get the hang of matching CGI objects with synthetic cameras , there is then the whole other art of "blending" ... seamlessly adding synthetic objects so that the viewer cannot easily tell what is real ... and what has been carefully crafted .

A FEW MORE OF MY REALSOFT3D TUTORIALS