What exactly are you trying to capture?

Is the image rectangular, does it have any distinct features, what kind of context will it be placed in, is this full-screen accelerated display or desktop image, is the image even in system memory, or is it rendered via overlay?

Then, you need to determine how to match the pixels. Will the screen be normalized or color-corrected? Can you match on per-pixel basis, or will you need to first transform into feature space? How will you build your comparison database for fast lookups? How will you check for false positives?

What you want is far from trivial. Not just from machine vision perspective, but can also be extremely hard from technical perspective. Graphic card manufacturers offer special hardware based solutions for this since there's so much processing involved.

what do you mean by transparent pixel? once stuff is rendered to the screen there is no such thing as transparency. Everything just has a simple RGB value, that's it. What's displayed to the screen is just a 2D grid of RGB values.

Image recognition is very hard, hard as in this is something that some of the best programmers in the world are working on but can't do well yet except for under specific circumstances.

So what _exactly_ are you trying to find on the screen? If it's exactly the same image displayed at exactly the same size every time, this is relatively easy. If it's something like "every time my friend Bob is on the screen" it's very very hard.

[EDIT: and to understand what I mean by "very hard": http://fishbowl.pastiche.org/2007/07/17/understanding_engineers_feasibility [smile]]

-me

[Edited by - Palidine on August 4, 2007 3:48:19 PM]

Quote:So what _exactly_ are you trying to find on the screen?

This isn't just a question from pesky doubters.

This is the crucial thing that will affect everything: the way you capture the screen, which algorithm you use, how to transform between captured image and format you want, how to locate the 64x64 square, and everything else.

*All* computer vision works on an incredibly rigid set of rules. Step only a bit aside, and it will fail miserably.

It's the very fact that everything is just a RGB value that makes this task so hard. If you had access to original data, or some other form of context, then it's easy.


Feature space transform takes RGB image and does something to it. For example, if you were looking for rectangles on screen, you'd apply Hough transform on image and obtain feature space consisting of nothing but lines. Then you'd post-process these lines to detect intersections, and figure out which of these comprise a rectangle.

Almost without exception all machine vision is a set of transformations, that turn RGB pixels into feature space that you can use.

This is why it's critical to specify, to last detail, exactly what you're trying to do, from which a suitable algorithm can then be chosen.

These are some basic example of how this looks in real-life, along with all the side-effects you get even under controlled conditions.

The print screen functionality is user-invokable just search msdn for the hook. Otherwise I believe SDL gives you that functionality as do DirectX and OpenGL.

Quote:Original post by Peter Conn
The only differences will be the position and the background - by transparency I mean that the image I am searching for has some transparent pixels in it - which won't be drawn to the screen, therefore I will just skip checking for those pixels.

Remember that "same size" means same size in pixels. It will only work if the displayed resolution of the image is exactly the same as the resolution of the sample image. i.e. if the sample image is 100x100 pixels, this algorithm only works if the displayed image takes up exactly 100x100 pixels on the screen. So if it's appearing in an application that scales the image relative to window size, or relative to screen resolution this doesn't work. [smile]

That implementation is something like:



If resolutions change or do not match sample image, or sample image rotates or skews, or gets partially occluded by other windows, etc, prepare to write at least a few hundred or thousand more lines of code. =)

-me

Well given your description this shouldnt be too hard. You mentioned that the image is always exactly the same and is never rotated.
So basically just can the entire image top to bottom and left to right. Once you hit a pixel that is equal to the pixel of the "image you are searching for"'s top left corner increment an index counter. Then check the next pixel in your scan against the pixel one to the right of the corner and so on. If they dont match once reset back to the top left corner and continue. Extending to 2d isnt hard.

Oh and:
1.) Use any language, up to you.
2.) I think there is DevIL for c++ which ive never used, also there is the Win32 API and DirectX could help. Java has built in help and c# probably does too.
3.) you can use directX for screenshots, there may be another way I dont know.
4.) Id suggest minimizing it, check the msdn.

Regards

-CProgrammer

http://opencvlibrary.sourceforge.net/

OpenCV is a library designed for similar problems. Collegues of mine currently use to to achiefe optical tracking functionality. It´s meant to be fast.

No warranty it's suited for your personal situation.

gl