I am working on GUI for my game, but I think this question could apply to any object in general - How to clearly highlight object regardless of its color? Currently I am blending its result color with other semi-transparent color(light blue), but obviously - this does not work for blue objects.
What are my options if I would like an object highlighted? I am looking for OpenGL shader solutions, best in customizable ones (e.g. more variants/ intensities).
You could create some kind of contrast.
Like:
Black
Blue
Black
Sorround the GUI object with that and its pretty clear what youre pointing at.
But this is only one way of doing this.
There is a trick in, I think, one of the Graphics Gems books: drawing an high-contrast overlay on any background by flipping the most significant bit of each colour component of the current frame buffer colour. (Yes, it could be done in OpenGL.1.0/2.0 without shaders.) Increasing or decreasing each colour component by half the maximum value provides more than enough contrast with adjacent unaltered pixels.
This technique might look bad with unpleasant frontiers between high and low values (slightly noisy backgrounds with components around half amplitude explode into spotted patterns) or simply because the overlay echoes the same texture as the background.
If you can afford to prepare graphics more carefully you might simply draw something normally, fully opaque, with a double outline: one dark and one light. The contrasting outlines can't both be confused with the background, and they form a high-contrast edge between them. A double outline can often be simplified to a dark outline with a light fill or vice versa, particularly in the case of text in solid colours.
An interesting way - I would not thought of that. 
However, I probably misled you in my original post. I am dont technically need to be in contrast with background(game) - the button's color/texture I will choose manually. My thoughts were, if I once go through all the GUI implementing, I will probably use it in my next game too. But that also mean, the button can be of other color/texture in the next game and I would like to have some nice MouseOver effect on it - regardless of its color.
Do you really need an automatic approach? I would think that having different "themes" for each UI would be the best approach. In my UI library, I have implemented the possibility of parsing UI themes from a XAML-like file. This way you can specify all the UI appearance without needing to recompile everything. You can have a look at this post in my journal.
Well, technically I dont. And the more I think about it, the more I realize I probably dont need it. I just wanted to have some default GUI style for quick prototyping that would work OK anywhere.
It is interesting - this is exactly what I am trying to achieve with OpenGL. Reinventing the wheel and creating cross-platform XAML based GUI for my games. Actually I think there is no good free GUI for OpenGL games (Yes, I know there is CEGUI, but I tried it and I did not like it - I dont think I really need 1500+ classes for GUI) at all.
I assume you are using C++ otherwise I'd be open to the idea of making it cross-platform. But at the moment I only have the time to continue its SharpDX/D2D development. Feel free to have a look anyway, the code is freely available on github.
If we're talking about buttons and such, one possibility would be to use greyscale for disabled, desaturated for normal, and fully saturated colour for highlighted. This will always work irrespective of the current background.
Swizzle the color channels ? Do it cyclicly (over time ) so it stands out by its constant changing ??
The grayscale for disabled is great, but I would like to stick with "normal" for normal. Is there some kind of vivid manipulation?
@wodinoneeye can you link some example of result? It is quite hard to imagine how it looks like.
