Sign in to follow this  
lmbarns

Augmented Reality

Recommended Posts

lmbarns    460

Firstly, I'm trying to learn more about tracking objects in a video stream.

 

I have the camera rendering on a texture in a game engine. So far I've been thinking I need to screen capture each frame, or every few frames, and look for matching pixels to track the x, y position of. So if you're tracking a white paper on black surface, as you move it around I can track the position and rotation...

 

 

Anyone have any insight to the topic? I've seen there are numerous expensive products and API's, I was just thinking it shouldn't be that impossible to track a simple square or rectangle sheet of paper on a contrasting background, and spawn models on it, adjusting them to the rotation/position of the paper as it moves...

 

Thanks.

Edited by lmbarns

Share this post


Link to post
Share on other sites
n0name    225

You must consider what happens if you rotate your camera... what if you move it away or closer to the object you're trying to track... what if lighting changes....

This are  all factors that affect the way pixels look... so just comparing the pixels themselves isn't a good idea.

Usually what you compare is some correlation between the pixels

Also before you start tracking your object you'll need to find it.

Read about SURF, SIFT, BRIEF algorithms ... they might give you some insight...

 

If you have a static camera and just move the piece of paper ... you should be able to get away by just finding the corners of the paper by using a feature detector

and then on every frame just calculate features on small area arround each paper corner and compare.... 

Share this post


Link to post
Share on other sites
VladR    722

You don't need those expensive products.

 

In my previous job, we have used OpenCV which is free.

 

 

Getting it to compile under Linux environment that you can't really break or properly configure is no fun, though - prepare to switch between different versions of gcc during the build, handling different versions of ImageMagick (and no, certainly not the latest, that tend to explicit various bugs in OpenCV).

 

Should you desire to compile OpenCV with the GPU support, there is even more fun in front of you :-)

 

Once you get through the incredibly problematic linux compilation, it is easy to use.

 

You can start looking into SURF algorithm, of which there are lots and lots of papers on the internet.

Share this post


Link to post
Share on other sites
lmbarns    460

Thanks guys. I've found a few implementations with OpenCV and Unity game engine. SURF looks promising.

 

If I use unity it seems the tracking will have to be via plugin from something much faster like cpp.

 

Unity has a method WebCamTexture, which I was using to render the material on a cube. Then I was testing out just finding matching pixels using WebCamTexture.GetPixels32() which gives me a byte array of the pixels of the entire texture.

 

Looping through the pixels I check each for a match with the color I'm looking for. If it finds it, I divide the pixel index by the width of the texture, and am able to get the x, y coordinate where the pixel was found on the texture.

 

The other thing I was thinking just for matching the colors, if I add several tiny rects that try to find the corners and just check them rather than grabbing the entire array of pixels making up the entire texture, just check small rects.

 

I'm aiming for just for a static camera and a non moving paper first, then I'll go from there with movement, looks like it can get really insanely complex quickly.

 

I was testing it on my face and can use the eyedropper to select a pixel then scan for matching pixels and it finds a decent clump, like if I eyedrop the white of my eyes it finds the white on both eyes. Or a piece of skin on my face finds a cluster around it.

Edited by lmbarns

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this