Jump to content
  • Advertisement
Sign in to follow this  
Karl Strings

Would anyone like a nice IO Completion Ports example?

This topic is 3106 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

For anyone who polls this board, I have a rather nice and clean example of using IO completion ports with sockets to make an echo server and I am using it for interviewing right now. Obviously I am not going to need it once I am done and I would not mind posting most of the code (it's a lot of code) and a short description on what is going on. This would be very useful to someone starting out with server code because it touches on some very complicated topics like lock free algorithms, synchronization, look aside lists... This is not something that someone outside of a professional programming environment would be likely to see. I want to post this because all the completion port examples I have seen on the net are either fundamentally broken, have improper/broken synchronization, or just very bad code - at one point in my life I wasted a lot of time with those examples... The lack of any good examples makes me nervous, I am not sure if anyone would actually take the time to review some very complex C code and actually see how completion ports and the surrounding synchronization, buffer handling, and lock-free algorithms are correctly implemented into a working system. So... Would anyone be interested in seeing this code or will I just be wasting a lot of my time doing a write up and post? Please post if you would like to see it. One rule: If you post includes the request to translate this code into C++/C#/Java or porting it to Linux please don't post. -Karl Strings

Share this post


Link to post
Share on other sites
Advertisement
I think this is a great idea. Good examples are always helpful, especially when it is clearly stated why they do things in a certain way (plus, perhaps, a note on why it is wrong to do the same thing in a different way).
Plus, I've been wanting to learn IOCP for a while now :D. You wouldn't want to leave somebody salivating for a great example and not getting it, would you?

Share this post


Link to post
Share on other sites
I'm also interested in it, I want to learn IOCP too, and its true that is difficult to find good information and code examples in the net.

Share this post


Link to post
Share on other sites
Personally, I have no more use for such an example (apart from being curious, you never know whether you can still learn something from someone else's code), but it is certainly something that would be very valuable. It might save people who are still struggling on that particular thing several weeks of pain.

It might be even more helpful if it included file i/o as well, and a Linux epoll port. Completion ports are much easier (almost trivial, once you have waded through all documentation) to get right with only sockets, and the same is true for epoll. Getting both sockets and file read/writes into one "thing" without blocking, however, is rather non-trivial.

Share this post


Link to post
Share on other sites
Quote:
Original post by samoth

It might be even more helpful if it included file i/o as well, and a Linux epoll port.


Or, instead of dealing with API pecularities, just focus on actual tasks.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Quote:
Original post by samoth

It might be even more helpful if it included file i/o as well, and a Linux epoll port.


Or, instead of dealing with API pecularities, just focus on actual tasks.


This has been an area of contention for me for quite some time. In most applications abstracting the OS with a library is a good thing. It lets developers concentrate more on what they are doing and less on what the underlying OS is doing. This changes when you enter the arena of system and/or server software. Modern operating systems provide lots of support for high performance applications and in order to take full advantage of that support one must intimately tie the application to the OS. You need to be just as concerned with the underlying OS and what it is doing as well as what you are doing.

Adding general purpose abstraction layers so close to the hardware does make your life easier, but you lose a key advantage. The vast majority of the time that advantage does not matter (super performant code on a quad core 2.4ghz machine gains very little for user applications.) and you can use C#.NET, Java, Boost, whatever your flavor is, to its fullest potential and be just fine. For pedagogical reasons, its best that people learn what is really happening on the metal. It keeps people from writing 10,000 element linked lists that have to be iterated though every 100ms because it is easy to write "LinkedList<MyClass> LList; LList.Find(Whatever);" without understanding the consequences of big O and cache poisoning.

Anyway, it seems as if some people are interested so I will do a write up and post once interview season is over.

PS. The MSDN example for AcceptEx uses IOCP, and it is completely broken. If anyone is looking for a good IOCP QA exercise, go to http://msdn.microsoft.com/en-us/library/ms737524(VS.85).aspx and figure out why it is broken. If you post your answer here I will let you know if you are right or not. It's right on the surface, no digging required, so most people should be able to find it just by looking at the code sample long enough, and I doubt throwing the code into a debugger will be any help.

-Karl Strings

Share this post


Link to post
Share on other sites
Quote:
Original post by Karl Strings
This has been an area of contention for me for quite some time. In most applications abstracting the OS with a library is a good thing. It lets developers concentrate more on what they are doing and less on what the underlying OS is doing. This changes when you enter the arena of system and/or server software. Modern operating systems provide lots of support for high performance applications and in order to take full advantage of that support one must intimately tie the application to the OS. You need to be just as concerned with the underlying OS and what it is doing as well as what you are doing.


Why would that be necessary when abstraction layers such as Boost.Asio which Antheus suggested simply provide wrappers around these high performance OS facilities? One may argue that the abstraction layer might be slightly slower than calling the native API directly, but knowing the Boost community I would say the performance loss is negligible. Besides, that sort of reasoning can be taken just as far as you like: "The C++ compiler generates slow code. I'm better off writing the assembly code by hand."

Quote:
Original post by Karl Strings
Adding general purpose abstraction layers so close to the hardware does make your life easier, but you lose a key advantage. The vast majority of the time that advantage does not matter (super performant code on a quad core 2.4ghz machine gains very little for user applications.) and you can use C#.NET, Java, Boost, whatever your flavor is, to its fullest potential and be just fine. For pedagogical reasons, its best that people learn what is really happening on the metal. It keeps people from writing 10,000 element linked lists that have to be iterated though every 100ms because it is easy to write "LinkedList<MyClass> LList; LList.Find(Whatever);" without understanding the consequences of big O and cache poisoning.


In a sense, I agree with you on this. It is important to understand your tools. However, whether or not it's worth your time really depends on what your goal is. If you are trying to gain a deeper understanding of the technology you are working with (in this case, Operating System specific scalable IO), playing around with that is obviously the way to go. However, if you're making a game, you will probably be better off using a reliable abstraction layer written people who know what they are doing (such as the Boost developers). That way, you just might even finish your project. ;)

A few days ago I ran across a thread on an electronics forum where a guy presented his idea for making an ADC (Analog-to-digital converter) from discrete components. The replies were generally discouraging, and the consensus appeared to be that "10 years ago, it might've been fun; today, you're probably better off making something with an ADC." Once again, making the ADC from discrete components would enable him to learn something. Building something else using an ADC IC would allow him to accomplish something.

Share this post


Link to post
Share on other sites
Quote:
Original post by Windryder
Why would that be necessary when abstraction layers such as Boost.Asio which Antheus suggested simply provide wrappers around these high performance OS facilities?


My reply was regarding "linux and file handling". Not the rest of thread or OP.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Quote:
Original post by Windryder
Why would that be necessary when abstraction layers such as Boost.Asio which Antheus suggested simply provide wrappers around these high performance OS facilities?


My reply was regarding "linux and file handling". Not the rest of thread or OP.


From your post I did get the feeling that you encourage people to apply this principle in a more general context ("focus on the actual tasks"). Hence I thought it appropriate to refer to your suggestion in my post.

Share this post


Link to post
Share on other sites
Quote:
Original post by Windryder

From your post I did get the feeling that you encourage people to apply this principle in a more general context ("focus on the actual tasks").


If someone wants to study IOCP using plain C, that's fine.

But when it comes to cross-platform portability and related abstractions, the problem is solved satisfactory by asio or ACE. And reinventing the wheel in that particular case is somewhat counter-productive.

Portability, especially in C and C++ is a tedious nightmare of typedefs and macros, and lots of obscure debugging around undocumented or other edge cases, and studying something like that for purpose of learning is, IMHO, a waste of time. It's just one of those things that requires years-long tenure of hands-on practice and simply cannot be taught from some book or source code tutorial.

Or, at very least, it's probably best to start with some existing abstraction, then drill down as to why certain choices were made the way they were.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!