Jump to content
  • Advertisement
Sign in to follow this  
janta

Win32 asynchronous I/O

This topic is 4158 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hey everyone, one quick question. I have the following code:
OVERLAPPED overlapped;
ZeroMemory(&overlapped, sizeof(OVERLAPPED));
ReadFileEx(hFile, pBuffer, 1024, &overlapped, Callback);

// do other stuff...

// And now...
... I need to wait until the file read has completed. The problem is there may be other async I/O request that have been issued, so just putting the thread in an alertable wait state wont do the trick since I need to be sure *that particular* read has completed. Technically, what I'd like to do is have a hEvent and to the following
Callback()
{
    SetEvent(hEvent);
}

// as long as the read is not complete, keep listening:

while(WaitForSingleObject(hEvent, 0) != WAIT_OBJECT_0)
    SleepEx(INFINITE, TRUE);
But that just loks ugly and I'm sure there's a better design arround this (Or do I have do design such as there aren't several async I/O requests pending at the same time?) EDIT: Maybe I should just get rid of the event and just use a simple bool instead ? I think I've been fooled into usin an Event since such exist in the OVERLAPPED structure, but actually here I don't think I even need that...
Callback() { bDone = true; }

// as long as the read is not complete, keep listening:

do
{
    SleepEx(INFINITE, TRUE);
} while(!bDone);

Thanks [smile] JA

Share this post


Link to post
Share on other sites
Advertisement
My apologies if I'm misunderstanding your problem, but if you have to wait until the read is finished then why don't you just use synchronous I/O instead?

Also, if you choose the bool approach, don't forget to protect the bool with some form of mutex (or at the very least flag it as volatile). That said, I see nothing wrong with having the completion routine signal an event to wake up your other thread. Why do you think that's ugly?

Share this post


Link to post
Share on other sites
You do realize that the OVERLAPPED structure has an event handle inside of it that you can set to an event you create that you can THEN wait on...right?

I would also recommend reading up more on IOCP and Asynchronous IO, as you are missing some fundamentals in how to use it.

See GetOverlappedResult.

Share this post


Link to post
Share on other sites
Quote:
Original post by Red Ant
My apologies if I'm misunderstanding your problem, but if you have to wait until the read is finished then why don't you just use synchronous I/O instead?


In my real code I'm trying to process a file using a double buffer scheme, so I do something like

0 - Initialize 1 buffer (synchronous read)
1 - Start reading next buffer (Async)
2 - Process current buffer
3 - Wait till nex buffer is ready
4 - swap buffers and repeat 1 until done

Quote:
Also, if you choose the bool approach, don't forget to protect the bool with some form of mutex (or at the very least flag it as volatile). That said, I see nothing wrong with having the completion routine signal an event to wake up your other thread. Why do you think that's ugly?


I dont see the need of synchronization for read write operations on the boolean value since everything happens on the same thread (the Callback called when IO operation completes is also called on the same thread)

I may have misunderstood something though... Thanks for your reply !
JA

Share this post


Link to post
Share on other sites
Quote:
Original post by Washu
You do realize that the OVERLAPPED structure has an event handle inside of it that you can set to an event you create that you can THEN wait on...right?

I would also recommend reading up more on IOCP and Asynchronous IO, as you are missing some fundamentals in how to use it.

See GetOverlappedResult.


Sorry, as I understands it, that does not fit my needs. My real code is more complex and I need some processing to be done when the read completes, so I do need that callback...

Ok, here's what I'm actually trying to do.
Pleas keep the discussion about bad coding style/effectiveness/reinventing the wheel/whatever on the side. I'm just experimenting and having fun here ;)



// -------------------------------------------

void CALLBACK UTBufferedFileStreamCallback(DWORD dwErrorCode, DWORD dwNumberOfBytesTransfered, LPOVERLAPPED lpOverlapped)
{
UTBufferedFileStream* pStream = (UTBufferedFileStream*) lpOverlapped->hEvent;

pStream->muiFilePos += dwNumberOfBytesTransfered;

pStream->mNxtBuffer.uiValidSize = dwNumberOfBytesTransfered;
pStream->mNxtBuffer.uiReadPos = 0;
pStream->mNxtBuffer.bIsValid = TRUE;

if(pStream->muiFilePos < pStream->muiFileSize)
return;

pStream->muiFilePos = pStream->muiFileSize;
}

// -------------------------------------------

bool UTBufferedFileStream::SetSource(const WSTRING _pFileName)
{
DWORD dwCreationFlags = FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED | FILE_FLAG_SEQUENTIAL_SCAN;
mhFile = CreateFile(_pFileName, GENERIC_READ, 0, 0, OPEN_EXISTING, dwCreationFlags, 0);

muiFilePos = 0;
if(mhFile == INVALID_HANDLE_VALUE)
return false;

muiFileSize = GetFileSize(mhFile, NULL);

ZeroMemory(&mOverlapped, sizeof(OVERLAPPED));
mOverlapped.hEvent = (HANDLE) this;
mOverlapped.Offset = muiFilePos;

mNxtBuffer.bIsValid = FALSE;
ReadFileEx(mhFile, mNxtBuffer.pBuffer, muiBufferSize, &mOverlapped, UTBufferedFileStreamCallback);

PrepareNextBuffer();

return true;
}

// -------------------------------------------

void UTBufferedFileStream::PrepareNextBuffer()
{
// Wait for the previous read operation to complete (= wait for mpNxtBuffer to be ready)
do
{
SleepEx(INFINITE, TRUE);
}
while (!mNxtBuffer.bIsValid);

UTExchange(mCurBuffer, mNxtBuffer);

ZeroMemory(&mOverlapped, sizeof(OVERLAPPED));
mOverlapped.hEvent = (HANDLE) this;
mOverlapped.Offset = muiFilePos;

mNxtBuffer.bIsValid = FALSE;

if(muiFilePos < muiFileSize)
ReadFileEx(mhFile, mNxtBuffer.pBuffer, muiBufferSize, &mOverlapped, UTBufferedFileStreamCallback);
}

// -------------------------------------------

uint32 UTBufferedFileStream::Read(void* _pDest, uint32 _uiBytesToRead)
{
uint8* pDest = (uint8*) _pDest;

uint32 uiBytesReadSoFar = 0;

// SOMETHING IS WRONG HERE, I KNOW
for (;;)
{
uint32 uiBytesToRead = _uiBytesToRead - uiBytesReadSoFar;
uint32 uiBytesRead = mCurBuffer.CopyTo(pDest + uiBytesReadSoFar, uiBytesToRead);

uiBytesReadSoFar += uiBytesRead;

if ( (uiBytesRead == uiBytesToRead) || (uiBytesRead == 0))
break;

PrepareNextBuffer();

}

return uiBytesReadSoFar;

}




This code still has many bugs, I just posted it in an attempts to make people understand where I'm trying to get at.

Regards,
JA

Share this post


Link to post
Share on other sites
Quote:
Original post by janta
I dont see the need of synchronization for read write operations on the boolean value since everything happens on the same thread (the Callback called when IO operation completes is also called on the same thread)



Ah yes, you are correct. I just read up on that, and yeah it's how you say it works. I was under the impression that an asynchronous call was really nothing else than one thread scheduling a routine to be run in another thread and that thread then notifying the thread that requested the async call. Now I see it's handled using interrupts instead. Sorry for the confusion.

Share this post


Link to post
Share on other sites
The overlapped structure is passed straight through to the callback routine. As such you can define a new structure that contains an Overlapped structure and any extra information you need, including for instance a marker to wait on that particular read until it completes (set an event and have the completion routine toggle the event when it's fired).

Also, your this pointer is not an OS handle. Don't set the hEvent to it (even though it IS safe to do).

Share this post


Link to post
Share on other sites
Quote:
Original post by Washu
The overlapped structure is passed straight through to the callback routine. As such you can define a new structure that contains an Overlapped structure and any extra information you need, including for instance a marker to wait on that particular read until it completes (set an event and have the completion routine toggle the event when it's fired).

Also, your this pointer is not an OS handle. Don't set the hEvent to it (even though it IS safe to do).


Well I'm 100% aware of this, what let you think I was using my "this" as an OS handle?...
I'm just following Microsoft's guidelines, which you seem to be aware of as well

Quote:
The ReadFileEx function ignores the OVERLAPPED structure's hEvent member. An application is free to use that member for its own purposes in the context of a ReadFileEx call. ReadFileEx signals completion of its read operation by calling, or queuing a call to, the completion routine pointed to by lpCompletionRoutine, so it does not need an event handle.

(From MS's ReadFileEx documentation)


And I think I'm doing just what you advise me, in the callback routine:
pStream->mNxtBuffer.bIsValid = TRUE;


to signal that the read operation to this buffer is done, though I'm missing the reason why I'd need an event rather than a simple boolean. Could you explain that part please ?

JA

Share this post


Link to post
Share on other sites
Quote:
Original post by janta
Quote:
Original post by Washu
The overlapped structure is passed straight through to the callback routine. As such you can define a new structure that contains an Overlapped structure and any extra information you need, including for instance a marker to wait on that particular read until it completes (set an event and have the completion routine toggle the event when it's fired).

Also, your this pointer is not an OS handle. Don't set the hEvent to it (even though it IS safe to do).


Well I'm 100% aware of this, what let you think I was using my "this" as an OS handle?...
I'm just following Microsoft's guidelines, which you seem to be aware of as well

Quote:
The ReadFileEx function ignores the OVERLAPPED structure's hEvent member. An application is free to use that member for its own purposes in the context of a ReadFileEx call. ReadFileEx signals completion of its read operation by calling, or queuing a call to, the completion routine pointed to by lpCompletionRoutine, so it does not need an event handle.

(From MS's ReadFileEx documentation)

Indeed I am aware of that. However that is not recommended by microsoft, just a note that you CAN use it.
Quote:

And I think I'm doing just what you advise me, in the callback routine:
pStream->mNxtBuffer.bIsValid = TRUE;


to signal that the read operation to this buffer is done, though I'm missing the reason why I'd need an event rather than a simple boolean. Could you explain that part please ?

JA

You can't wait on a boolean (short of a loop). You can wait on an event, which will allow callbacks to fire, until the event is set by the callback you are waiting for. From what you've described, you may have a series of IO events in the queue executing, but you want to wait for a specific one of those IO events to complete before continuing. Your choices are then either a LOOP, or an OS event object. The former will consume CPU time with little benefit (unless you have secondary processing that you can perform), the latter will not consume any processor time except for the IOCP events that fire. This will allow for other threads to run.

Share this post


Link to post
Share on other sites
Err.. I feel kinda stupid coz I still don't get it.

At first I had tried to do the following, which is what you advise me, IIUC


callback() { SetEvent(event); }

// then to wait till the read is done:
WaitForSingleObjectEx(event, INFINITE, TRUE);


The problem I found is thta the callback sets the event to a signaled state, but the WaitForSingleObjectEx does not return for that reason but because an IOCP has been invoked, thus causing my event to remain in it's signaled state, and the next time I try to wait for read completion, WaitForSingleObjectEx returns immediately without actually waiting for anything (since it remained signaled...)

Sure I could create a manual-reset event and call ResetEvent() but that would allow just any IOCP to unblcok my WaitForSingleObjectEx(), which is not what I want.

So I eventually did the following:


do
{
SleepEx(INFINITE, TRUE); // Allows IOCP to be invoked
}
while(!bDone); // checks that the IOCP that I was waiting for has been invoked. Otherwise, start listening again.


How would you do that differently?

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!