Cadde

Well i don't know much about the history of DirectX... And despite some rumblings about the accuracy of Wikipedia it's at least a good start to get a brief overview of DX history.
There are two articles of interest, one covering DirectX and the other covering Direct3D.

Instead of me linking them just google "DirectX" and "Direct3D" and you will find them in the top 3 results.
Beyond that you might have to google each and every version of DirectX and Direct3D separately and build a history from there.

Unless of course you can find a game developer that has been around since 1992 and is willing to share his knowledge with you.

To answer your question "why it was necessary to" it's simply because demand exceeded the capabilities of the latest version and hardware grew ever faster and more capable. OpenGL played a big part in speeding up the development of DirectX and when OpenGL slowed down (version 1.3) so did DirectX (Version 9.0 a, b, c) and furthermore, the Xbox and PS3 can only use hardware compatible with DirectX 9 and OpenGL | ES 1.0 (OpenGL 1.3) with some features of OGL | ES 2.0 (OpenGL 2.0) respectively.
Only 'recently' (with the arrival of Vista, DirectX 10) an enormous demand has arrived for MORE. As gamers and developers demand ever higher fidelity in rendering (graphics) both hardware and drivers need to match the expecations.

So, simply put... It needed to evolve because we wanted more!
And there are many many more techniques to come as hardware grows, not necessarily in speed but in width.