a. Just that feeling while moving mouse and it doesn't give you the fluency (lol).
Ugh, measure the framerate. Possibly in milliseconds & (less priority) in fps.
b. Hmm, before I have posted the question I thought my tree has around 300 vertices that's why I was wondering why the game is not fluent with 300 trees, but actually it has 2800... I don't know how could I overlook this.
2800v * 300 instances = 840.000 vertices per frame.
At 30 fps that's 25.200.000 vertices per second. That's still way below what a decent GPU can perform (i.e. a quick Google shows that the old Geforce 6600 can do 375 million vertices per second, divide that value by 4 to 10 because
those are raw specs)
But of course, it's not the same if you're running in an Intel HD 3000 series than a GeForce 690 or Radeon HD 7990. What GPU do you have? (and what CPU btw.)
c. I don't think it's heavy. I calculate there instance data like position, scale and rotation + multiplying it by the primary matrices (world, view, projection).
Ok, fair enough. May be there's some inefficiency that has been unnoticed (i.e. I assume you pass the worldViewProj matrix, instead of passing the 3 matrices and concatenating them in the shader)
d. Alpha blend with black&white map.
e. No.
Gotcha! This is most likely the culprit. Alpha blending is expensive. It consumes a lot of bandwidth. Switch to alpha test. If you don't like the results, consider using alpha blending for the close trees, and alpha blending for the far trees.
Using CSAA for nice, smooth leaves while using alpha testing is also another possibility (though this topic is probably more advanced)
f. What is that? I am passing depth to the pixel shader in order to calculate fog.
Good, don't worry. Just checking. Btw. That feature allows you to alter from the depth in pixel shader that was going to be passed to Z Buffer that was calculated from the vertex shader. However this has severe performance impacts.
If you don't know what it is, then you're probably not using it.
g. No shadows yet. Don't even know what the cascades are.
Ok, just checking. Don't worry, you'll get there.
Cheers