Sign in to follow this  
janta

redirecting stdout and stderr to a single file

Recommended Posts

Hi guys I have a program that processes about 150,000 files and outputs one line of text for each file to the standard output stream, plus one line of text every time an error is found with a file, to the standard error stream. Oviously, the amount of information makes it impossible to read from a command prompt, so I have to redirect the output to a file. The bad thing is the error message does not reference the file it applies to, so it's like:
(stdout) processing file blah.txt
(stderr) error found!
I know how to redirect both streams to a file:
program.exe > log.txt 2>&1
but I think that because buffering, the messages are not reported in order which makes it impossible for me to track down the error. What I get is basically 150,000 lines of "processing blah blah blah", and then at the end a single line "error found!", which of course doesn't apply to the last processed file at all. I can not modify the software I am using (which is perforce, by the way) How could I deal with this? I've tried to google this but didn't find anything interesting. Thanks in advance for any help!

Share this post


Link to post
Share on other sites
What makes you think that the messages are in order when they're displayed to the console? And if they're not, what makes you think that the information about the correct ordering can be retrieved at all?

Share this post


Link to post
Share on other sites
Pr. Zahlman , I usually am a big fan of your maieutics, but could you make your point more explicit here? [smile]

Let me rephrase my question: is there a way to make sure that both standard and error streams of a single process are reported in the same order as they are written to.

The reason I think it's possible is because it makes sense to me. That does not mean it is actually possible. If I knew I wouldn't be posting here.

Best,

Share this post


Link to post
Share on other sites
Quote:
Original post by janta
Let me rephrase my question: is there a way to make sure that both standard and error streams of a single process are reported in the same order as they are written to.

The reason I think it's possible is because it makes sense to me. That does not mean it is actually possible. If I knew I wouldn't be posting here.


Expanding on Zahlman's post -

Consider the standard output stream and the standard error stream as two independent objects, which they really are. With these stream objects, there are certain internal buffering mechanisms that prevent output from always going directly out to the "console" (just for a simple example, it's a little more complicated under the hood). The console is a shared resource, so if at any time any three of the output stream objects use it, there will be some problems with the result (it has to do with I/O resources).

So if one object were to write "Hello World" and at the same time (i.e, when you are using threads) another object was to write "123456", then the actual output that you see is not defined. It could print the messages in order, it could print "Hello 12345World6", or any number of other combinations. If you want a simple analogy, think about a 2 lane one way road that merges into one lane. There is no way to keep the cars in the same original order they were traveling in.

If you just had one thread running, then output should always be serialized and you would avoid any problems with multiple writes. If you didn't though, there is no way to guarantee correct ordered output of the text unless the application had synchronization in place for accessing the output stream objects.

So, you don't have many possible ways to solve the problem. You could try:

1. See if you can make perforce use only one thread for processing all those files so output is serialized and you can follow all the input sequentially. I don't use it so I'm not sure how it's setup or even if this is possible, but I would imagine it is using more than one thread right now to process all those files. If it is currently using one thread at a time, you should not be having problems, so I think it's safe to assume it is! Likewise, look for any options related to this issue in the output format so it does not do this.

2. Try to use this MSDN article: How to spawn console processes with redirected standard handles, only if you can run perforce as a console application or invoke it through a console prompt via cmd line and hope you can at least dump the text that way. I would think that you will get the exact same output from your original stream redirection example, but it's always worth a shot for verification.

3. This is out of the question here I assume, but sometimes you can just patch the client yourself or inject your own DLL to detour the logging mechanisms and synchronize them yourself. I don't think it's a realistic solution for you here, but that's quite often one way to get around these problems.

I don't think there is really any easy way around it aside from reconfiguring perforce if it was possible. You might want to contact their support though as they might know of how to solve this issue if you have not already. Good luck!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this