I'm simply changing the values in the color array.
How are you doing this? Is it a VBO or a client-side vertex array? What hints was it created with?
the memory usage slowly decreases to around 600MB (after a couple of minutes).
Is this actually a problem? If the driver doesn't need that memory for other tasks right now, they it's ok for it to delay releasing it.