I'm co-designing a graphics engine which is supposed to support both DirectX (>=9) and OpenGL (>=2) with support for the fixed function pipeline.

DirectX and OpenGL use different coordinate systems (left-handed vs. right-handed), so care must be taken when dealing with matrices. Currently, I've got a function [font="Courier New"]set_transformation_matrix(matrix4x4)[/font] which flips the z axis in the OpenGL backend and doesn't in the DirectX backend.

However, I also want to support shaders. My shader class has a function [font="Courier New"]set_uniform_matrix(string name,matrix4x4)[/font] which sets a matrix. This setter function, however, should not flip the z axis per se, because we might want to use the matrix for something other than projection.

So my question is: How do I treat the coordinate systems in my engine? Or, if I were to abstract GLSL and HLSL, how would I treat projection matrices? "Tag" them with "is_projection_matrix"?

You're both missing the point.

I cannot simply "pick" a coordinate system, since the user should be able to set shader matrix variables himself. And I cannot just transform those matrices to match OpenGL's or DirectX's coordinate system since I don't know if the matrices are used for projection or something else. If they're not used for projection, the transformation leads to unexpected behavior.

If you can't pick the coordinate system, then you must let the user pick it and then call the API's LH or RH functions depending on the user's choice.

I cannot simply "pick" a coordinate system, since the user should be able to set shader matrix variables himself. And I cannot just transform those matrices to match OpenGL's or DirectX's coordinate system since I don't know if the matrices are used for projection or something else....

AFAIK at some point it must be ensured that the involved matrices match a convention before they are concatenated. There must be a decision to use the one or other system. If the user is responsible to set any and all matrices then you can leave this decision making to her anyway. Otherwise you need a prescription or a preferences setting.

... If they're not used for projection, the transformation leads to unexpected behavior.

This is not exactly true. Because the mirroring happens inside a chain of transformations, you can adapt either the left or else right matrix, because of
( P * M ) * V == P * ( M * V )
when using column vector notation where P means the projection, M the mirroring and V the view matrix. However, you're right in that you must not shift the mirroring onto the side of the model matrices as long as you don't want the user to be able to intermix handedness at will.

Ok, first of all, thanks for all your input. We've looked at each post and then decided on a solution, which is as follows:

The convention we use in the renderer is a left-handed system. The renderer provides projection functions for this system only (copies of the DirectX LHS projection functions). In set_transformation_matrix(matrix4x4 input), [font="Arial"]the OpenGL backend transforms the input matrix by doing[/font]

translation_matrix({0,0,-1}) * scaling_matrix({0,0,2}) * input

[font="Arial"]which accounts for the different canonical view volumes in OpenGL and DirectX. Thus, OpenGL also uses a LHS system, the z axis is not flipped.[/font]
[font="Arial"]
Additionally, the shaders now do not receive a plain [font="Courier New"]matrix4x4[/font] in the [font="Courier New"]set_uniform_matrix[/font] function anymore, but a new class containing the [font="Courier New"]matrix4x4[/font] and a flag indicating if the supplied matrix is a projection matrix. If that's the case, the same transformation as above is done.[/font]