In my opinion, "engine" is a terrible word to describe whatever code you write to help you build a server.
In general, an "engine" is a complete set of integrated game development subsystems. This includes physics, sound, input, graphics, AI, networking, etc.

I suggest that you figure out what it is you want to do, first! Do you want to write a networked game? If so, write a networked game!
Do you want to write code that other people can use to write networked games? Then download existing products that do something similar, and play around with them, to figure out how this could be done. Then make your own spin on it. (Also, you may want to articulate in some detail what's wrong with the existing solutions, first)
Do you want to learn networked game server programming? Don't worry about providing an "engine" or a "framework" or even a "library," because you're learning! Dive into some existing open source project or library, and learn from what already exists.

Personally, I kind-of like the "library" approach. Define some area of functionality you want to provide -- say, transforming data structures to bits in a buffer and back (marshalling), or matchmaking between players and available game sessions based on some various properties -- then implement a library that performs that function. Write sufficient sample code and documentation and tests that other people can actually understand and make use of it. Then find another area of functionality you need, and develop another library -- repeat until you feel you've covered what you need to cover!