Hi,

I've got a closed-source program, compiled without debug symbols, whose behavior I would like to tweak for my own personal use. Specifically, the program provides a network service that I want, but creates a window that I do not want; I would like to be able to launch the program invisibly at startup.

Presumably, the window is created by a call to either CreateWindow or (more likely) CreateWindowEx, in which case I just need to make sure that the 'WS_VISIBLE' bit in 'dwstyle' is set to zero (which can be achieved just by setting the whole DWORD to zero). So that's the one change I want to make: change the fourth argument to CreateWindowEx to zero.

Hence my first question is: If a program is compiled without debug symbols, is it still possible to identify when it is calling which Windows API functions? (for instance, can it be done using the debugger in Visual Studio 2008?) Is there a good way to locate the call to CreateWindow or CreateWindowEx? (Later on in this post, I'll describe my attempts to figure this out.)

Before continuing, let me note that ordinarily, I wouldn't go through all this trouble. The way I would normally run a program invisibly is to just call ShellExecute from a little C++ program with the 'nShowCmd' argument set to SW_HIDE. Equivalently (and this is what I actually tried), I'd use 'shell.Run' from the Windows Scripting Host with the second (intWindowstyle) argument set to zero, like so:

// start-server.js//// Invisibly launches server"use strict";var shell = WScript.CreateObject("WScript.Shell");shell.Run("server.exe -start", 0, false);

However, programs are not required to respect the intWindowstyle argument, and what is interesting is that although this program will listen if I tell it to run minimized, or maximized, or normally, it will not listen if I tell it to run invisibly. This is what leads me to modify the program itself.

So let's dive in.

I opened up the EXE in the Visual Studio debugger, and by stepping through it found exactly where the window is created - I think. I find the call trace,

00481397  call        0048F5D7   0048F5E7  call        00490590      004905F9  call        dword ptr [eax+50h]         004011A6  call        00490B1C            00490BF8  call        00493505               00493576  call        00493C9D                  00493CAB  call        dword ptr ds:[49E43Ch]   (server.exe)                     7E42AF60  call        dword ptr [edx]          (user32.dll)

where I have represented levels of nested functions with indentation. The innermost function call (I haven't gone any deeper) creates a window without any buttons, and appears to be being made by user32.dll. The preceding function call, which then seems to be into user32.dll, is made by the program itself. Hence, assuming I've interpreted things correctly, the line that I'm looking for is,

00493CAB  call        dword ptr ds:[49E43Ch]

This line exists in the function starting at 00493C9D, so let me show the code from the start of this functon to the function call (plus two lines afterward):

00493C9D  mov         eax,dword ptr [ecx+38h] 00493CA0  test        eax,eax 00493CA2  jne         00493CB3 00493CA4  push        dword ptr [esp+4] 00493CA8  push        dword ptr [ecx+1Ch] 00493CAB  call        dword ptr ds:[49E43Ch] <------------- CreateWindow(Ex)?00493CB1  jmp         00493CC1 00493CB3  mov         edx,dword ptr [eax] 

Now there seems to be a contradiction: Only two things are pushed on the stack prior to the call to ds:[49E43Ch], which would imply that it is a function that only takes two arguments. Both CreateWindow and CreateWindowEx take many more arguments than this!

Hence, if anyone has any insights for what might be going on here (and where I might have gone astray), I'd appreciate them too.

(Thanks!)

Compilers generate out of order code. Other variables can be set elsewhere, initialized before main even runs (even if not declared in such way, but are determined to be constant), or other means can be used to set it, not necessarily via esp.

Since there is no stack cleanup in example the calling convention might be stdcall.

There's a debugger option for Visual Studio called Load DLL Exports which you want to enable. You should also set up the Microsoft Symbol Server. It'll be much easier to debug with those on as you'll see some function names.

For example with those on in disassembly view you can go to the CreateWindowEx function (with Control + G) and add a breakpoint there to find out what calls it.

By the way I suspect if you look further up the code you'll find where the other parameters are pushed. However if they were parameters to the function you're looking at the compiler in some cases won't bother getting them off the stack to push them back on again.

The simplest way to find the parameter values is just to open a memory window and look at the data around esp at the time of the call instruction. Knowing the calling convention you can easily work out what they are.

Having said all that if you can put up with the window appearing very briefly instead of modifying the exe I'd suggest finding the window and hiding it from another program. I found one example of how to do that in Powershell which comes with source code.

Quote:Original post by Adam_42
Having said all that if you can put up with the window appearing very briefly instead of modifying the exe I'd suggest finding the window and hiding it from another program. I found one example of how to do that in Powershell which comes with source code.

Nice link; thanks. This had crossed my mind (albeit not using PowerShell to do it), but unless I get totally frustrated I'll stick with trying to modify the EXE, because (1) it simply feels more entertaining, and (2) it should give a slightly more "perfect" result (since the window will never appear, even for a split second). But I'll go that route if I decide to get practical. :-)

Quote:
There's a debugger option for Visual Studio called Load DLL Exports which you want to enable.

This by itself has been very useful! After turning this on I see that the function being called is actually ShowWindow, which does take two arguments.

Hence if I can change this

00493CA4  push        dword ptr [esp+4]      <------------- nCmdShow00493CA8  push        dword ptr [ecx+1Ch]    <------------- hWnd00493CAB  call        dword ptr ds:[49E43Ch] <------------- ShowWindow

to this

00493CA4  push        dword 0                <------------- nCmdShow (CHANGED)00493CA8  push        dword ptr [ecx+1Ch]    <------------- hWnd00493CAB  call        dword ptr ds:[49E43Ch] <------------- ShowWindow

I should achieve what I want, assuming I have interpreted the stdcall calling convention reference correctly. (It might also suffice to just replace those three instructions with NOPs.) Hence my first question is -- assuming this is correct -- how would I affect this change? Hex editor? Disassemble-edit-assemble?

Quote:You should also set up the Microsoft Symbol Server. It'll be much easier to debug with those on as you'll see some function names.

Haven't tried this yet; I will if it starts to seem like more information would help.

Quote:The simplest way to find the parameter values is just to open a memory window and look at the data around esp at the time of the call instruction. Knowing the calling convention you can easily work out what they are.

Ok, I tried this... At the time when my debug cursor is sitting at the 'call' (00493CAB), ESP = 0012FD14, and memory looks like this:

0x0012FCF4  ab ae 42 7e 64 00 00 00  «®B~d...0x0012FCFC  00 00 00 00 88 fd 12 00  ....ˆý..0x0012FD04  ab ae 42 7e 30 4b 4e 00  «®B~0KN.0x0012FD0C  68 35 49 00 30 4b 4e 00  h5I.0KN.0x0012FD14  3c 04 31 00 01 00 00 00  <.1.....  <-------- ESP (hWnd?)0x0012FD1C  7b 35 49 00 01 00 00 00  {5I.....  <------------ (nCmdShow?)0x0012FD24  00 00 00 00 88 fd 12 00  ....ˆý..0x0012FD2C  7c fd 12 00 04 00 00 00  |ý......0x0012FD34  01 00 00 00 00 00 00 00  ........0x0012FD3C  01 00 00 00 00 00 00 00  ........0x0012FD44  fd 0b 49 00 04 00 00 00  ý.I.....

This is odd, because I would expect nCmdShow to take one of the two-digit-at-most values given here. I was expecting to see
"00 00 00 00 00 00 00 01" somewhere, since SW_SHOWNORMAL=1. That makes the addresses 0x0012FD34 and 0x0012FD3C catch my eye, but I don't think I've screwed endianness up, have I? This throws some doubt on my understanding. Hence my second question is if anyone can spot a mistake I've made in my reasoning.

Thanks again!

Unless you're looking at 64 bit code, those arguments are going to be 4 pairs of hex digits, not 8. I think your value for nCmdShow is the second half of the 0x0012FD14 line, so it's equal to 01 00 00 00. Intel is little-endian so that's the hex you'd expect to see for 1 or SW_SHOWNORMAL.

Quote:Original post by Emergent

0x0012FD14  3c 04 31 00 01 00 00 00  <.1.....  <-------- ESP (hWnd?)0x0012FD1C  7b 35 49 00 01 00 00 00

HWND is the '3c 04 31 00'
nCmdShow is the '01 00 00 00'.

'7b 35 49 00' is 0x49357B which is the return address for the '00493576 call 00493C9D' instruction.
'01 00 00 00' is another copy of nCmdShow passed in.


All you really need to do is replace the instruction that pushes any of the copies of nCmdShow onto the stack (either the first one or the second one).

In Visual Studio, turn on "show instruction bytes" in the disassembly window so you can see the machine code for the instruction:

00493CA4 (machine code would be here) push dword ptr [esp+4]

You want to replace that with 'push imm32(0)'.

From the intel manual, the bitwise encodings are:

PUSH - Push Operand onto the Stack  register:    1111 1111 : 11 110 reg  register:    0101 0 reg  memory:      1111 1111 : mod 110 r/m  immediate:   0110 10s0 : immediate

The existing instruction is the 'memory' form, and you want the immediate form. The good news is that the immediate form takes fewer bytes than the memory form for your case (which will be using the Opcode, ModR/M and SIB bytes and at least one displacement byte), so you can replace it. The 's' in '10s0' is a wildcard bit which controls sign-extension. If the bit is 1, the immediate data is 8-bit instead of 32-bit (in this case the sign extension is like casting a char to an int, so the parameter will still take up 32 bits on the stack). Since you want to pass 0, you can use the 8-bit version to ensure your instruction fits.

So your encoded bytes for 'push 0' would be '6A 00'.

You will need to replace any unused bytes of the original instruction with single-byte NOP instructions:

NOP - No Operation:     1001 0000

So your replacement bytes would be '6A 00 <90 90 90 90...>' with however many 90s you need to fully replace the original push (which you can find out easiest by displaying the instruction bytes in Visual Studio)

[Edited by - Nypyren on June 19, 2010 2:03:52 PM]

SUCCESS!

(I'd still appreciate some explanation about the inconsistency in the debugger.)
[EDIT: While I was writing this, Nypyren and pinacolada helpfully replied.]

Microsoft ships a program called 'dumpbin.exe' with Visual Studio which is capable of disassembling programs; you run it like so:

dumpbin.exe /disasm program.exe > program.asm

I did this, and found the corresponding lines in it (the ones I talked about in my last post); they were

  00493CA4: FF 74 24 04        push        dword ptr [esp+4]  00493CA8: FF 71 1C           push        dword ptr [ecx+1Ch]  00493CAB: FF 15 3C E4 49 00  call        dword ptr ds:[0049E43Ch]

What is interesting here is that this gives you not just the ASM, but also the hex values of the machine code [EDIT: Nypyren pointed out that this information is also available from the debugger]. So I opened up the exe file in a hex editor (I used XVI32), and did a find-and-replace, to find

FF 74 24 04 FF 71 1C FF 15 3C E4 49 00

and replace it with

90 90 90 90 90 90 90 90 90 90 90 90 90,

which, Wikipedia tells me (assuming I understand correctly) is a string of NOPs.

When I run the modified EXE, no window appears (however, whatever other window is current loses focus. I suppose that if I had actually called ShowWindow with the SW_HIDE argument instead of NOPping it out, this loss-of-focus would not have occurred?)! However, the program appears in Task Manager, and I can connect to the network service it provides!

Apart from the extremely small aesthetic issue of the current window losing focus (which doesn't much matter at startup), this is exactly what I had set out to achieve!! I'm feeling a little 1337 right now. ;-)

The only task that possibly remains is to replace the NOPs by a proper call to ShowWindow with SW_HIDE. [EDIT: I've tried this; it works just as well as the NOPs, but the current window still loses focus. This is not a major issue!]

Also, apparent success aside, there are still a few things I haven't understood about what I just did (in particular, the apparent inconsistency between what I expected and what I saw on the stack), so if anyone can offer explanations I'd appreciate it! [EDIT: Nypren and pinacolada have explained this nicely. Thanks so much, guys!]

[Edited by - Emergent on June 19, 2010 3:24:56 PM]

Quote:Original post by Extrarius
In the future, if you want to reverse engineer a program, I'd strongly suggest using a tool made for that purpose. Two examples of such a tool are OllyDbg(free) and IDA Pro(free version is older edition and limited in what it will open, but is still a very good tool).

Thanks; those would have been helpful. I just tried playing around with OllyDbg (the ability to insert new assembly on the fly is awesome), and found some ways to refine my patch; there are some calls to EnableWindow and SetFocus immediately around the code I'd previously found that I can also NOP out which will fix the focus "problem."*

The server also includes a little message console in the GUI; maybe with these improved tools I can get ambitious and redirect it to a file (maybe I can even write some code in C and link it in somehow)... Of course, at this rate maybe I should have just written my own server, but this is more fun...

* [EDIT: ...or not. It looks like this code never even gets executed on Vista/Core2 (had previously been working on XP/PentiumM) and that now the window is just created by a call to CreateDialogIndirectA... or maybe OllyDbg just deals with threads differently from VS and it's getting executed there as a callback from a new thread launched by CreateDialogIndirectA, and my breakpoints are associated to the wrong thread... whatever; I'll figure it out if I decide I want to spend more time on it...]

[Edited by - Emergent on June 20, 2010 9:12:38 AM]