The way you use vertexData (allocate it with new, then add stuff to it, then subtract what you added to it before calling delete vertexData) is just asking for trouble, just make a copy of the pointer if you are going to manipulate it.
I'd do something like this:
byte* vertexData = new byte[bufferSize];
float* floatArray = reinterpret_cast<float*>(vertexData);
// do stuff with float array, no need to keep adding sizeof(float) either when you work with floatArray, just use ++floatArray. Don't ever change the value of vertexData
EDIT: Whoops ;)