AN INTRODUCTION TO VIDEO CAMERA MATCHING WITH RS3D
Garry Curtis January 28 2007 A tutorial for Realsoft3D V5
Skill Level Suggested :
Another tutorial in the "Saved From The Abyss" Series , which are tutorials
concerning little known but very useful information that may have very briefly appeared on
the Realsoft3D Mailing List .
There is a ton of entertainment shown today on television
and of course, cinema , that use a blend of live action and computer generated content .
The art form is so seamless in it's presentation it is almost impossible to determine
what is which (well , the 2 headed aliens are more than likely CGI) or exactly what
part of the room or outdoor shot is real , and how much of it is done with 3D
computer S/W ?
RS3D provides many elaborate tools designed to easily render
synthetic scene objects at a level of photorealistic quality far beyond what is required
for typical video insertion . The 3D Software industry seems to be devoted to realism , yet ,
in fact , one of the easiest way's to "sell" your highly realistic renders is to simly place them
into a video or photograph , in such a manner that the CGI objects are actually enhanced
by their realworld surroundings ! Really !
Anyway , hopefully this tutorial will let you in on how
to begin to experiment with combining video and your very own RS3D CGI , using
freeware S/W's , like Icarus or Voodoo and Virtualdub . Realsoft has provided several
import plugins for importing realworld camera data (via camera tracking software)
, and automatically creating an exact synthetic version of any camera inside RS3D .
This tutorial will not deal with 'Image Aspect
Ratio' or the RS3D camera . I highly recommend you visit
primer for info on that topic after you've done this
tutorial , as a means of better understanding some very important issues you
will sooner or later have to come to terms with .
The skills involved with matching shadows or lighting
will be left to another time , perhaps . It is already quite a
lengthy "One-Page Tutorial" and really is only about getting data into
Realsoft3D , and how to match a RS3D object with that imported data .
Once again I should also mention
that it written for the beginner , thus the rather long-winded manner .
Apologies to the advanced RS3D users who must muddle through the
basics in their effort to learn something new .
A Simple Realsoft3D Example
Here's what we are aiming for in this very basic
our resulting animation (Xvid 400KB)
**NOTE** - Here's another
recent example (Xvid 4MB)
That is , to blend a RS3D 3D Object with a 2D Object from the
Voodoo camera matching S/W's
included Free-Move example . Select a slow play speed in MS Media Player
or view it in
in a frame by frame manner , or at whatever speed you desire via the play controls .
The above animation came from the RS3D project file
included in this .zip file
Grab the .zip file and load the project into RS3D . The .jpeg's are included , and
all that may be required is to direct the paths to the project's materials .
Load it and do that now , have a look around inside the project , then
save and close the project and start the tutorial , which will begin with
focusing on the actual camera tracking Software ... Icarus .
Icarus has been defunct for years now , since it's
technology was bought out , reworked and is now being sold as a
Pixel Farm Product .
Icarus can still be used for non-commercial
and educational uses , but not for profit . Google still shows a couple of places on
the net that have it available for download .
In all fairness to you and to the S/W developers , I
should mention that RS3D does import several other commercial camera tracking
packages , and since I haven't really used any , I can't vouch for them here .
I have seen the SynthEyes demo
SynthEyes's Web Site .
and found the program to at least look very comprehensive and appears to be
a bargain at $399 USD . Excellent tech support and an active forum as well .
SynthEyes does have a very good example of a camera
flyover of a pyramid object , with an output script in a format that is written
specifically for RS3D . It is java based and looks to be an excellent "one-step
handshake" between it and RS3D . The SynthEyes Method seems to provide a
truly professional solution . Download the
image sequence & camera output script from the SynthEyes site , and then
import both into RS3D and judge for yourself .
This tutorial and Realsoft3D project is based on another freeware and open
sourced camera tracking software - Voodoo . RS3D does not import Voodoo output
but we will use one of the Voodoo S/W installs image sequences , (a large
yellow tractor) since it is a very good example of large camera movements
around a well defined subject .
LET'S TRY IT !
OK , now that the introduction is over with , let's get
quickly into this very simple process of creating camera data and then importing it into RS3D .
We'll go through these steps in a general way that may apply for any video or
image sequence , not just the included .jpeg image sequence . Do attempt to match
my results using Icarus by creating your own Realsoft3D camera and see how
it compares to the Icarus export in the above-mentioned .zip file .
Icarus is a little picky about what kinds of formats it can import .
If the video or animation that you want to use is in an image format , such
as .jpeg or .bmp , or if it's an mpeg , open
and load that image sequence and export it as an uncompressed .AVI
(for Icarus in the next step , and for RS3D , later) . Uncompressed
is always best ... no compression artifacts to confuse the camera tracking S/W .
Export the .BMPs as an .AVI
Open Icarus by double clicking on Calibration.exe in the "Icarus/bin" folder. Create
a new Project (Project/NEW) and import the .AVI (Project/Import Movie) . The Camera Parameters
Control Panel will open automatically . Icarus default is 'Free Motion' and 'Constant Focal' .
Change 'Constant Focal' to 'Fixed Focal' (using the pulldown menu) and then choose your
type of camera from the pulldown list (see image below) .
Then , type in your camera's focal length . Leave everything else as is and choose
'Close' . If you have a video with unknown source then just leave these parameters as is for
now . If the camera that Icarus builds for you does not seem to be working in Realsoft3D
(or if the exported 'Features' do not match as well as Icarus indicates)
you may have to come back to this Camera Parameters control and try again . Choose
'Constant Focal' instead of 'Fixed' and let Icarus try again for you . You may
also have to experiment with various CCD sizes if you are working with a difficult
If Icarus is having trouble with a video clip , try running the whole video
segment through , not just an edited clip . The more frames the better . As a
last resort for problems with determining proper Focal Length , try running
the video through the freeware "Voodoo" camera tracking software . It has
more modern algorythms and seems to solve easier , as does Syntheyes .
Voodoo Web Site
I should mention that for best results we should always use a video with a constant
focal length (no zooming) . If you have a video that contains zooms you are free to
attempt a matching by choosing the 'Variable Focal' control (from the pulldown menu)
but I can not recommend it simply because I've never attempted it .
Your Focal Length Control Window
Under the Tracking Menu , select 'Set Start Frame , then move the slider ,
to the last frame of the sequence and then 'Set End Frame' . Move back to
the first frame and then under the Camera Menu , Select 'Track and Calibrate' .
Icarus will look at possible points (Features) in your video to track using the default
settings . Allow Icarus some time to do this .
OK , it's done tracking possible feature points so we can have a quick look at the data
points that have are showing in the View window . Notice how Icarus has placed bright green
points on prevalent positions . You should see the exact same thing inside RS3D .
Let's get a feel for what Icarus has done ... Select 'View Window' from the 'View' pulldown
menu and a new window will open showing you the Icarus camera and the 'Feature' points
that have been determined .
Pan and move a zoom around in this view and see if any of the points seem to portray a
pattern that may loosly resemble objects in
our video . Hopefully we will be able to pick out an object or 2 , but if not ,
don't worry , it's not critical .
One more quick look at some other data and we are ready to export the camera . At the far
left on the vertical panel are the Project's Parameters . Select the Camera Parameter
object and expand it . Then select and expand the other parameter boxes . Select any
of the Camera 'Translation' or 'Rotation' elements and you'll notice a graph will appear at
the very bottom .
This wavy line represents our camera's path through 3 dimensional space , as well
as the camera's movement on it's own internal axis . Now , before we leave Icarus
we want to check for 3 things . Select the 'Focal Length' element from this parameter
list and we should see a straight line in the Graphing Window at bottom . Remember ,
we selected 'Fixed Focal' so this line must be straight , which tells us that the
camera's focal length did not change during the course of the video .
Do the same for 'Aspect Ratio' and 'Skew' . These should also be straight-line
(constant) data . Same for our Principal Points on the X and Y axis . If these
particular parameters are not straight , then we will have to re-do the camera
tracking and calibrating . Checking for this now will prevent a lot of
possible time-wasting inside Realsoft3D while trying to figure out why things
are not matching properly .
Your Project's Parameters
We can finally export the tracking data
and those feature points in a file that can be imported into RS3D . Select , from the 'Project'
menu , 'Export 3D Motion' , and then when the file saving control opens , under 'File Type'
scroll down to 'Human Readable (.txt)' . Make sure the Auto Features is set to
'ALL' , then choose a directory and a file name and click on 'OK' . This will export the
3D position of the camera , as well as the 3D position of the scene's prominent 'Features'
. Close Icarus .
Export the Data
THAT PART IS ALL DONE .
Now we can start RS3D and simply load the human readable
text file . I prefer to work from a top view , so choose the top view
and make sure you are using an 'Orthographic View' (not a 'Perspective' one) .
Use the general 'File/Load' command , or go down to 'File/Import/Icarus'
instead . An Icarus loader control will pop open and present you with an option
to size the scene . I also
prefer to load at "Scale" 0.10 , but for now just use Scale 1.0 and unselect 'Replace' .
Import the Data
Icarus Import Loader
Don't look for the Icarus data points until you have
done 2 things . First , right click in the
view window and go down to "Camera/Tracking" and make sure it is unselected . This way
we can move freely and not have to worry about accidentally being forced to
look through the lens of the 'Icarus Camera Object' , at least not until we have a good
look at this scene from a top view .
Make Sure Tracking Is Unselected
Second step is to hit the rewind button on your
'Animation Control Bar' . If you don't already have one open ,
open one now and hit rewind and make sure we are at the very first frame
(Frame #0 by default) . You probably won't see much yet , so
select the Icarus "Features" object in the "Geometric Objects Tab"
(first Tab in the Select Window) and "Drag&Drop" it
into your viewport . Hit the space bar to enter edit mode . We won't edit them
but your Feature Points will now have their "Red" edit boxes visible , making
them much easier to see .
The Icarus Scene Features
Now , zoom way out (or drag & drop the project's 'Root'
into the viewport window) and then select the "Icarus" object in the
"Geometric Objects Window" and observe the viewport window . A group of handles just jumped to
the location of this , the Icarus Camera Object . Zoom in until
you can easily make out what the Icarus Camera Object looks like from above , and where it
is pointing to (the features , of course) .
The Icarus Camera Object
LET'S GET TO WORK .
We may be getting the feeling that the scene is a fairly
large one . Actually , this particular scene is not too bad , as we can see if we
"Drag&Drop" the "Root" object into the 'Viewport' , but some scenes can be huge,
depending on how the features have been laid out by Icarus . Some
feature points could be hundreds of meters away . We can delete them later ,
or we could have chosen to just export certain features from
Icarus . You'll get the hang of that later , but for now let's just try and get
a "feel" for the size of this test scene .
If you are zoomed into the camera in a top view ,
zoom out until you can see the camera and also a fair number of Features . Now ,
create a Camera (Tools/Creation/Create Camera) for this view . This will be
our working view (or one of them) , so rename this camera as (Work-Top) and
save the File as (File/Save As ) " Icarus-test-1 " . Feel free to create other
camera's , and even some objects that will help you see where certain features
are located . For instance , we could place a large rectangle under the main
group of Feature Points .
If we look at the layout of features (from above) we may be able
to distinguish a shape , roughly where the large Tractor was sitting in that
sequence of images that Icarus tracked . For now , let's place an Analytic Cube there
amongst that collection of Feature Points that seem to form a square-ish looking
angle (that is actually representing the tractor ) .
Create a cube , rotate it , and scale it so that it roughly fills those FP's (Feature Points)
then take a side view . You will probably have to extend the Cube's Depth .
Do so in this side view via the Cube's Property Window , or with the Cube's Handles .
Build Our First Object
If you're fairly happy with the Cube's size and placement
(remember , this is just a test) then "Drag&Drop" the Icarus Camera Object into the
View Window . There ... we are now looking through a CGI camera that Voodoo and Icarus decided
must have been used to film those tractor images . How does your new Cube look ?
Of course it's impossible to tell , without those tractor images there to compare it to .
Let's Add The Background Video
Yes , we are certainly missing one key ingredient here ... the
.TGA image sequence . We will apply that image sequence as a Camera Backdrop
, so select the Icarus Camera and then from the Menu , choose "Tools/Creation/Create
Backdrop Image" . In the requester that appears , in the 'New Image' field - select the
.TGA image sequence
we loaded into Voodoo , or the uncompressed .AVI that we loaded into Icarus .
Set it's distance to 450 , so that it will be visible , but not in the way or
blocking the view of our new Cube . Click "Accept"
The All Important Backdrop Image
RS3D will have automatically created a new shader for you
that will apply the .TGA's or the .AVI to the new 'Backdrop' for you , as the animation
plays or as you scrub the timeline . To be assured this is going to happen smoothly for
you , click on the second Tab in the Select Window , the Material Tab , and salect our
new 'BackDrop' shader (the very last shader at the bottom of the Tab) .
The Backdrop Shader
Double click on our new 'BackDrop' shader , then Click on the
'Advanced' Box & on 'Preview' (both at
the Top Left of the Shader Control Panel) . An image should appear at Top
Right , and a few lines of code in the 'Shader Control' area , below . The line of code
'illumination=Texture(mapcoords)' represents our backdrop images . Select that line
and then click on the 'Animation Tab' and change the Animation Type from 'None' to
'Automatic' . RS3D will instantly change the necessary corresponding fields ,
also automatically for you . Now that's all ready to go , so go back to
the Geometric Objects Tab of the Select Window .
Using OPENGL For Our Backdrop
First make sure that you are using an OpenGL canvas
and not just a wireframe view . Right Click in the View Window and at the very
bottom , select 'View Properties' . On the View Properties Control Window
that appears , select the 'Draw Tab' and in the 'Canvas Options' section ,
select 'OpenGL Shaded' .
You had previously Dragged the Icarus Camera Object
into the View window , so now you should see the image (or first frame of the .AVI)
of the tractor in the Viewport Window . Drag that camera in again if required .
The image is a very blurry OpenGL rendering , so let's fix that now .
Double click on the Backdrop Object (that's in the
Icarus Camera Level Object) and select the "SPEC" (Special) Tab in it's Property
Window. We can see that it is set for 450.000 , and if we need to adjust that
number later , this is the place to do so . Now , select the Wire Tab
and change the Texture Quality from the default "64" to "512" .
Wow , big difference !
Backdrop Shader's Texture Quality
Congratulations ! You're Almost There
Well , OK , how does your New Cube look now ? Adjust it
with the Numeric Inputs that are on the 'Spec Tab' of it's Property Window , until it
looks ... not bad (or so) . Now , let's take that quantum leap ! In the View Window ,
right click and go back to "Camera/Tracking" , and this time make sure it is selected.
Now , scrub through your 21 frame Timeline by grabbing the Timeline Slider .
Does your cube kind of seem to move in a manner that is
relative to how the tractor seems to move ? Probably not , at least not on your
very first try , so Drag & Drop your Top View Camera Object into the View window
and zoom in on your cube (switch back to wireframe mode for best visibility) .
Zoom right in and place your cube so that it lines up better , with the rough
approximation of the Feature Points representation of the back of the Tractor .
Take a side view and tweak the cube some more . Repeat
these steps until you think you can see the cube is moving in a similiar manner as the tractor
is in the images . As you may have guessed , your cube is not actually moving at all , but
only appears to be doing so because of the Icarus Camera's movement . The tractor
also is not moving , (nor the Backdrop) but only appears to be .
A Final Test of Proof
Let's Use The Smokestack's Features
Let's leave the cube (make it wireframe invisible or delete it)
, and as a final test , let's try and replace the tractor's
exaust pipe with a RS3D synthetic version . Zoom into the FP's from your overhead
camera's vantage point , until you begin to see something like the image above .
I've placed a long cylinder at this group of 6 FP's (feature points) which
Icarus has captured in it's tracking data . We should note that Icarus just happened to capture
this exhaust pipe as a series of feature points . However , I could have manually selected these
points , or even created these points inside Icarus , for export , to be included
in the Human Readable text file (see the Icarus documentation) .
Do the same and create a cylinder , then take a side
view and tweak the cylinder some more , stretching it out and moving it up or
down until it seems to sit amongst the FP's , where it should . Drop the Icarus
camera into the Viewport and then move , size and rotate the cylinder
(only by it's handles) , until it is exactly covering up the tractors
exaust pipe . Now scrub through the timeline again , slowly and frame-by-frame ,
carefully watching to see if the exhaust pipe is covered properly in every frame .
The Exhaust Pipe at Frame #10
If your cylinder is "drifting" , sometimes to the
right of the exhaust pipe , and then sometimes to the left , you probably do
not have the cylinder at the proper Z-depth (distance from the camera) ,
but that is easy to fix . Just take your overhead camera view of this area
and move the cylinder back (away from the camera) , then scrub the timeline again .
If it is better now , move it back more , if it is worse , then just move it
forward (toward the camera) and try again . When you think you're close do
an anim preview , or render out . The 'File/Render Window' is set to go , but
you will have to choose a file directory for the output animation .
That's about it . It is actually a matter
of trial & error ... practice , really . In order for a person to get a good feel for ,
what has often been called "The Black Art of Camera Matching" , one must
just keep practising , it's that simple . With Icarus , it's important to
either have a lot of frames or even better , frames with a lot of movement
to help determine what the camera's parameters are .
Truthfully , this is just one step along the way .
Once you do finally get the hang of matching CGI objects with synthetic cameras ,
there is then the whole other art of "blending" ... seamlessly
adding synthetic objects so that the viewer cannot easily tell what
is real ... and what has been carefully crafted .
"Cheers !" & Best of Luck With Your Work .