I'm trying to do basically do point cloud alignment, but aligning to a specific plane. I'm generating a heightmap form the points, but they're slightly sloped, with that i mean, the ground plane isn't on the x and y plane, but rather sloped up. so i want to align it back down, in order for the heightmap to not be rotated by a certain angle. i calculated my eigenvectors and ended up with -0.8703 -0.0393 -0.0905 0.0000 -0.3573 -0.6427 0.0002 -0.3027 -0.6972 but then i'm stuck, i don't know what to do next in order to do what i want. i've tried multiplying all 3d points with a matrix made up of the eigenvector, and that didn't work, i tried multiplying them by a 3x3 rotation matrix created by using the 1st and 2nd eigenvectors and this code,

float** MathUtil::calculateAlignmentRotMat(float* a, int al, float* b, int bl) {

    float angle = acosf(dot3(a, b));
    float* rotationAxis = cross3(a, b);

    return calculateRotationMat(rotationAxis, angle);
}

float** MathUtil::calculateRotationMat(float* rotationAxis, float angle) {
    float **result = new float*[3];

    for (int i = 0; i < 3; i++)
        result = new float[3];

    float rx = rotationAxis[0];
    float ry = rotationAxis[1];
    float rz = rotationAxis[2];

    delete[] rotationAxis;

    float cosAngle = cosf(angle);
    float sinAngle = sinf(angle);
    float c = 1.0 - cosAngle;

    result[0][0] = cosAngle + rx * rx * c;
    result[1][0] = rz * sinAngle + ry * rx * c;
    result[2][0] = -ry * sinAngle + rz * rx * c;
    result[0][1] = -rz * sinAngle + rx * ry * c;
    result[1][1] = cosAngle + ry * ry * c;
    result[2][1] = rx * sinAngle + rz * ry * c;
    result[0][2] = ry * sinAngle + rx * rz * c;
    result[1][2] = -rx * sinAngle + ry * rz * c;
    result[2][2] = cosAngle + rz * rz * c;
    return result;
}

but that also didn't work. my general apply function is

    rotMatrx = util->calculateAlignmentRotMat(vecRot[0], m, vecRot[1], m);

    float* interm2 = vector(m+1);
    float* tempm = vector(m+1);

    /* Form projections of row-points on first three prin. components. */
    /* Store in 'realData', overwriting original data. */
    for (i = 0; i < realN; i++) {
        for (j = 0; j < m; j++) {
            interm2[j] = realData[j];
        }
        interm2[3] = 1;
        /* data[j] will be overwritten */
        for (k = 0; k < 3; k++) {
            for (k2 = 0; k2 < m; k2++) {
                tempm[k2] += interm2[k2] * rotMatrx[k][k2];
            }
        }
        realData[0] = tempm[0];
        realData[1] = tempm[1];
        realData[2] = tempm[2];

    }

I really hope someone can help me out with this, Thanks Phyx

I don't fully understand your question. Can you try to clarify exactly what it is you want?

In a nutshell, this http://i43.tinypic.com/27xo6c5.png but in 3d.

for(each point)
point.z = (constant)

Will that do?

I don't think that'll work, mapping all the points to the same constant Z will in effect flatten the pointcloud, losing any and all structural information i had would it not. In the picture i had, the line is rather the direction of the point cloud. the after picture still has the points in the same location relative to this direction vector.

Try computing the Least Squares Plane and then transform the points by the inverse transform of the plane.

Ok so your diagram didn't show what you wanted :D

In that case you have no choice but to do a coordinate transformation as _swx_ suggests. Find the basis vectors and origin point for the plane that the point cloud is based on; this will give you a 4×4 transformation matrix m of the type:

( ....x.... 0 )( ....y.... 0 )( ....n.... 0 )( ....o.... 1 )

... where o is the origin point of the plane, n is the normal and x and y are two perpendicular basis vectors within the plane. (If your plane is only a little deviant from the target space, I recommend you make these vectors x=n×Y and y=X×n respectively, which will align them with the target's X and Y axes as closely as possible.)

Similarly, find the basis vectors for the plane onto which you which to project the point cloud, which make up a matrix M:

( ....X.... 0 )( ....Y.... 0 )( ....N.... 0 )( ....O.... 1 )

It is likely that your target axes X, Y and N will be the world axes [1,0,0], [0,1,0] and [0,0,1], making this a simple transformation matrix, but they do not have to be.

Now, to project a point p out of the coordinate space defined by m, and into the coordinate space defined by M, simply multiply by the inverse of m and then by M:

P = pm-1M

Ah, ok, thanks _swx_ and Bob Janova , yeah i do beleive my world axes are [1,0,0], [0,1,0] and [0,0,1]. and i think i already have m, so but i was only applying it to my points, so that might be why it was screwed up. I'll try the one you mentioned. Thanks :)