Archived

This topic is now archived and is closed to further replies.

Executing memory

This topic is 5374 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi, I''m just writing an in-game decompression library. Some DLLs will be in a .zip-file and I''m wondering whether I need to decompress them on the harddisk first or if it is possible to extract them direct into memory and execute them. I already did some searches but I didn''t find anything. Thanks for your help. BlackHC I do know that I don''t know anything.

Share this post


Link to post
Share on other sites
On Windows, if the exe is statically linked with those dlls, then they will need to be unpacked to disk before the exe can be executed. If the exe dynamically links with those dlls, then the OS needs to access those dlls on disk in order to load them into the address space of the process via LoadLibrary.

But, if you really want to dig into things just the same, perhaps you might find SoftWire helpful. Or perhaps FuBi.

Share this post


Link to post
Share on other sites
That''s doable, but entails writing your own PE loader (you''d have to do everything LoadLibrary does). Probably overkill.
You could instead pack them with UPX, at the cost of a 1.5 KB decompressor stub, and possibly lower compression ratios.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Why all this talk about UPX? Why on earth would anyone want to use it (and similar programs) to make programs and libraries smaller??

Disk space is cheap, man. RAM is more expensive. Packed executables will use 15% or so more RAM thah the same executable unpacked.

Share this post


Link to post
Share on other sites
UPX and similar executable packers are good for download-and-run-once programs like installers and demos but shouldn''t be used for everything - and certainly not for DLLs with code that could otherwise be shared. While I have still to measure the memory overhead for real programs, I suspect it could be considerably larger than 15% in many cases.

Also, filesystem-level compression is an option for many users with concerns about on-disk executable size.

Share this post


Link to post
Share on other sites
> Why all this talk about UPX? Why on earth would anyone want to use it (and similar programs) to make programs and libraries smaller?? <
Quicker download.

> Disk space is cheap, man. RAM is more expensive. Packed executables will use 15% or so more RAM thah the same executable unpacked. <
True. However, increased mem usage on the order of 15 kb is laughable.

quote:
UPX and similar executable packers are good for download-and-run-once programs like installers and demos but shouldn''t be used for everything - and certainly not for DLLs with code that could otherwise be shared.

Agreed. I don''t know what these DLLs are for, nor how big they are.

> Also, filesystem-level compression is an option for many users with concerns about on-disk executable size. <
Yes. Again, though, I don''t think that''s a problem

Share this post


Link to post
Share on other sites
I think the idea is to treat code and data in a similar way. While it's a nice idea, it's impossible to implement in a practical way with native win32 API. It would certainly be nice if LoadLibrary could take a pointer to a buffer filled with code, but we're not given that option .

If you look into .NET such a feat is not only possible, but can be used in practical ways in real world situations.

[edited by - CoffeeMug on March 26, 2003 1:37:56 AM]

Share this post


Link to post
Share on other sites
all you need to do to "execute memory" is to make instruction pointer point at the beginning of that memory block, after making sure that appropriate pages have execute access right. you can allocate a piece of memory from the heap and execute it; in fact, that''s how ATL implements HWND to CWindow mapping.
quote:
Original post by CoffeeMug
It would certainly be nice if LoadLibrary could take a pointer to a buffer filled with code, but we''re not given that option .


if you already have the code in memory, there''s no point in "loading" it anywhere, you can simply execute it from wherever it is (again, assuming you have the permissions).

Share this post


Link to post
Share on other sites
niyaw: it''s not that simple. You can load files with .com extention like that and execute them. I actually did that when I was experimenting with buffer overflows But loading a dll involves a bit more then that. For instance, modifying the the jumps in the code to account for the shift of the base address where the dll is loaded.

Share this post


Link to post
Share on other sites
of course, but that doesn't have anything to do with executing a piece of memory. the process of loading a dll is completely separate from the process of executing code in that dll.

i feel out of place now. probably shouldn't have posted in the first place.

[edited by - niyaw on March 26, 2003 3:02:20 AM]

Share this post


Link to post
Share on other sites
niyaw: he specifically asked about loading dlls from zip files

Executing a piece of code is of course as easy as this:

    
void RunCode(void *pCode)
{
_asm
{
jmp pCode
}
}


Or something to that extent (I'm not up to par with x86 assembly lately).

.NET offers an interesting ability to load assemblies from a memory buffer. It's probably not very useful in most situations, but one of the advantages is being able to treat code as a resource in a resource management system. I'm not sure about the performance of this feature but it shouldn't be slower then more conventional dynamic linking.

I don't suppose it would be *too* hard to write a LoadLibrary function that accepts a pointer to a buffer. It certainly involves a lot of work but it's not something open source community couldn't handle If you search long enough on sourceforge you'll probably be able to find a loader like that.

[edited by - CoffeeMug on March 26, 2003 3:40:48 AM]

Share this post


Link to post
Share on other sites
Iirc, the Feb 2002 edition of MSDN mag contains an article that describes how LoadLibrary and friends do what they do - ie how the system loads a pe file. PE files are described in a different article in that edition. Both articles are worth the read, imo.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
quote:
Original post by Jan Wassenberg
UPX and similar executable packers are good for download-and-run-once programs like installers and demos but shouldn''t be used for everything - and certainly not for DLLs with code that could otherwise be shared.

Agreed. I don''t know what these DLLs are for, nor how big they are.

> Also, filesystem-level compression is an option for many users with concerns about on-disk executable size. <
Yes. Again, though, I don''t think that''s a problem

I think the exe compressors try to solve a nonexistent problem (file size) and/or fails to solve another problem (software cracking)in the first place.

1) As a cracking countermeasure?[/B]

Congratulations, you probably added five minutes to the time it will take to crack your program. Click & go unpackers exist for almost all compressors. The few that don''t have such things can be dumped from memory while it''s unpacked (in which case you might have added 30 minutes to the time a crack for your program will be released).

[B]2) Compressing the size to save disk space?[/B]

A bloated EXE of 1 MB would be considered a small file in general today.

[B]3) Compressing to save download time of a run-once program?[/B]

While you could probably do this, and get better compression ratios than by using regular archivers, downloading 300kB vs. downloading 400kB is a non-issue to most people. You probably don''t have exe files with a size on a scale where it would make any greater difference. A installation program with all the content to be installed "built in" could be quite large, but then again, the content (which takes the largest space in a big, solid installation-exe) is already compressed with the installation program''s compressor, and compressing it further will only compress the installer''s stub program. This could even end up making the resulting file bigger, as the compressed content will be difficult to compress any further, yet the exe compactor has to insert compression metadata here and there.

Ok, now to the cons of using exe compressors....

Dirty pages is probably the most obvious disadvantage. Regular programs will have their executable code in memory pages that are marked "read-only", while a compressed EXE obviously must have its code in memory pages that can be written to.

Now, in Windows each process has its own virtual address space of 4 GB (out of which 2 are reserved for the OS but that''s another story...). To save physical memory, windows will map "clean" pages to the same physical memory locations. That is, if you run several instances of explorer.exe, physical memory will only be allocated once for the static parts (code, resources, etc) - the, read-only pages. Memory that is written to will obviously have to be allocated per-process and cannot be mapped to the same physical location.

If you''re compressing DLL files it gets even worse. If you compressed, say user32.dll, all programs that use that file would need to load their own copy of that library into physical memory. This can really add up.

All this doesn''t mean that compression doesn''t matter on single-instance applications. Not so. Consider the situation when windows runs low on memory and has to free up some memory to be able to fulfill another memory request. If a page is "clean", it can just be discarded. When an application then tries to access a discarded page, windows will read it in directly from the executable file, transparently to the application (possible due to the nice pagefault mechanism of IA32). If the page is dirty, windows cannot just discard it. It will have to write the page out to the swap file. And then read it from there when it''s needed again. Disk read vs. disk read AND write.

Now to the myths of exe compressors...

It is sometimes said, that since hard drives are slow, and the processor is fast(er), it will be faster to load a compressed EXE, since the OS has to read less from disk. In some cases this can be true, but often it isn''t.

In DOS the entire executable is loaded from disk to memory before it starts running. This is not the case with windows. In windows the executable is loaded as a memory mapped file. The program will be loaded "on demand" as it runs. The compressed executable otoh has to be loaded all at once. Else it won''t be able to decompress itself.

Share this post


Link to post
Share on other sites
quote:
Original post by Anonymous Poster
3) Compressing to save download time of a run-once program?

While you could probably do this, and get better compression ratios than by using regular archivers, downloading 300kB vs. downloading 400kB is a non-issue to most people.


For this specific case I think the advantages may outweigh the disadvantages. The user gets a slightly faster download without having to unpack the file manually, and the provider gets a slightly smaller bandwidth bill. The memory footprint of a run-once program is probably irrelevant. There are other ways of getting those benefits, but those may not be available for everyone.

For the OP: your best bet is to install the DLLs alongside your EXE in the first place rather than trying to extract them into memory from a zip file. That''s much easier and less error-prone, and not likely to be less efficient.

Share this post


Link to post
Share on other sites