Let's review again, at the beginning it is 46%. When I do free(colors) it jumps to 41%. Next, when I do vertices.clear(), it jumps to 26%.
Now, why is it still taking 26%?
Your method of tracking memory usage within your application sucks so badly, that it is not giving you useful information. There are better tools to do this.
Who knows why it's taking 26%. It could be a driver. It could be a memory leak in your code. It could be for any number of reasons.
If I only construct my vertices and remove them all the memory is 6%, However, If I keep colors and vertices, send them to VBO with the code below and then release them all, it still stays at 26%, why? is it something else that is consuming the memory?
It's possibly consumed within a driver. But once again. Your method of tracking memory usage is not useful. There are better tools (intrusive preferably) that can help track allocations within your app. Use one of those instead of getting stuck in a loop focusing on something of little to no importance. If it runs cleanly through valgrind, you really haven't got a problem. If your driver is consuming memory, then it's also fair to assume that the behaviour is specific to your driver / video card make. There's a good chance other drivers behave differently. The problem though, is that attempting to validate what is happening in a 3rd party compiled binary will not help to improve your code whatsoever. Validate YOUR code, and move on when it works.