Sign in to follow this  
ExErvus

DLL access mechanism between multiple applications

Recommended Posts

I have ignorantly used libraries for as long as I can remember, even developing my own. I would always statically link without really questioning what is happening and the differences between static and dynamic linking. So I finally decided to take some of my spare time and take a closer look.

 

As quoted from msdn:

 

"Many processes can use a single DLL simultaneously, sharing a single copy of the DLL in memory. In contrast, Windows must load a copy of the library code into memory for each application that is built with a static link library"

 

and

 

"Many applications can share a single copy of the DLL on disk. In contrast, each application built with a static link library has the library code linked into its executable image as a separate copy."

 

So the question is, how is this possible? It sounds like a race condition between applications instead of threads. If you are dynamically linking to a DLL and a single copy exists in memory, it does not make sense to me how multiple applications can simultaneously access the DLL and somehow manage the state of the guts of the called functions.

 

 

Share this post


Link to post
Share on other sites
Overly simplified explanation with some exceptions and special cases omitted for simplicity's sake:

So, the DLL consists of code and some read-only data that are shared (basically the same stuff that it has as a file on disk is what can be shared by multiple processes in RAM).

The dynamic allocations it makes when loaded in a process are NOT shared (and these allocations are also the vast majority of a process's memory consumption, at least for games and "big" programs).

In cases where the DLL's shared memory might be modified by one process, it uses https://en.wikipedia.org/wiki/Copy-on-write to give that process a unique writable area so that other processes are not affected.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa366785(v=vs.85).aspx
 

DLLs are created with a default base address. Every process that uses a DLL will try to load the DLL within its own address space at the default virtual address for the DLL. If multiple applications can load a DLL at its default virtual address, they can share the same physical pages for the DLL. If for some reason a process cannot load the DLL at the default address, it loads the DLL elsewhere. Copy-on-write protection forces some of the DLL's pages to be copied into different physical pages for this process, because the fixes for jump instructions are written within the DLL's pages, and they will be different for this process. If the code section contains many references to the data section, this can cause the entire code section to be copied to new physical pages.


Gory details: At compile time, DLLs and EXEs have some pointers which point to other areas within the EXE/DLL. These pointers are set as if the EXE/DLL were located at a specific position in memory (in the above MSDN quote, this is the "virtual default address" they mention).

Now, since EXEs and DLLs are all compiled separately, none of them know what the others' default virtual address is. So it's possible that when you go to start your process and load all your DLLs, some will have conflicting defaults (meaning, if it just loaded them all, some would overlap in memory). When this happens, it's first-come-first-serve, and DLLs that load later need to be put in a DIFFERENT position in memory instead. This requires "relocating" the DLL - all of those pointers that were created inside the DLL at compile time have to be updated to point to where the DLL is ACTUALLY loaded.

Since this involves updating those pointers, it can no longer be shared with different processes. So it makes a fresh copy of the DLL in that one process's memory so that it can relocate it without affecting any other processes.


TL;DR: DLL physical pages can be shared unless they are relocated, in which case a new set of physical pages is created for that case. Edited by Nypyren

Share this post


Link to post
Share on other sites

Overly simplified explanation with some exceptions and special cases omitted for simplicity's sake:

So, the DLL consists of code and some read-only data that are shared (basically the same stuff that it has as a file on disk is what can be shared by multiple processes in RAM).

The dynamic allocations it makes when loaded in a process are NOT shared (and these allocations are also the vast majority of a process's memory consumption, at least for games and "big" programs).

In cases where the DLL's shared memory might be modified by one process, it uses https://en.wikipedia.org/wiki/Copy-on-write to give that process a unique writable area so that other processes are not affected.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa366785(v=vs.85).aspx
 


DLLs are created with a default base address. Every process that uses a DLL will try to load the DLL within its own address space at the default virtual address for the DLL. If multiple applications can load a DLL at its default virtual address, they can share the same physical pages for the DLL. If for some reason a process cannot load the DLL at the default address, it loads the DLL elsewhere. Copy-on-write protection forces some of the DLL's pages to be copied into different physical pages for this process, because the fixes for jump instructions are written within the DLL's pages, and they will be different for this process. If the code section contains many references to the data section, this can cause the entire code section to be copied to new physical pages.


Gory details: At compile time, DLLs and EXEs have some pointers which point to other areas within the EXE/DLL. These pointers are set as if the EXE were located at a specific position in memory (in the above MSDN quote, this is the "virtual default address" they mention).

Now, since EXEs and DLLs are all compiled separately, none of them know what the others' default virtual address is. So it's possible that when you go to start your process and load all your DLLs, some will have conflicting defaults. When this happens, it's first-come-first-serve, and DLLs that load later need to be put in a DIFFERENT position in memory instead. This requires "relocating" the DLL - all of those pointers that were created inside the DLL at compile time have to be updated to point to where the DLL is ACTUALLY loaded.

Since this involves updating those pointers, it can no longer be shared with different processes. So it makes a fresh copy of the DLL in that one process's memory so that it can relocate it without affecting any other processes.

 

Good information indeed. Thank you.

 

So when talking about copy-on-write, and how applications share the dll as read memory, and create a new page of memory when writing to the dll, how does that translate to the actual content of the dll? Is that like separating logic flow from variables?

 

I guess to clarify, what would be a real example of an action inside an application that would trigger the OS to copy and create a new page of memory in the DLL?

Edited by ExErvus

Share this post


Link to post
Share on other sites

So when talking about copy-on-write, and how applications share the dll as read memory, and create a new page of memory when writing to the dll, how does that translate to the actual content of the dll? Is that like separating logic flow from variables?


DLLs contain the code, and a small subset of the variables that the program will actually use. Most variables are created on the stack, or on the heap. Both stack and heap are created dynamically and are not shared between processes. The operating system gives new processes their initial stack when they start, and then the code inside the program is free to request a heap and more stacks any time it wants.

The variables inside a DLL are globals (static variables) and read-only globals.
 

I guess to clarify, what would be a real example of an action inside an application that would trigger the OS to copy and create a new page of memory in the DLL?


The way copy-on-write works is the OS marks certain pages (the typically-4096-byte-chunks I mentioned in my previous post) with special flags using the VirtualProtect function. The flags can be set to cause exceptions to fire whenever the process read/write/executes from memory within that page. For copy-on-write, only write accesses cause the exception. Read and Execute work normally without any changes, since those are safe to do on shared memory.

The exception handler can then handle this by creating a copy of that page BEFORE the write actually happens, change the page protection on the new copy, and then allowing the write to continue where it left off, which now accesses the new copy instead.

Quick Summary: Any attempted write operation (of any kind) to a shared page triggers copy-on-write for that page.

This means that even if the DLL doesn't need to be relocated, if it has any global variables that get modified, the page containing that variable will be copied the moment the program first writes to it.

This stuff about triggering an exception when a page is accessed is built into the processor itself, and the OS handles it. If you're interested, you can search around for "x86 protected mode" and "x86 virtual memory" for even lower-level details (specifically the "descriptor table" stuff). Other processor architectures often have similar support that works in kind of the same ways.


Fun tangental note: The same page access exception stuff is used to detect and handle stack overflows! Edited by Nypyren

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this