Jump to content
  • Advertisement

Archived

This topic is now archived and is closed to further replies.

aftermath

Please help! DLL/EXE memory problem; very nasty.

This topic is 5738 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

this is sort of an advanced topic, so if you don’t know what a heap is or what cross-boundary memory sharing is, you probably won’t follow me on my dumb explanation. O.k. Here’s the beef. At the moment I am developing a 2D shoot'em'up for about 5 month now. I am also using my preparatory game engine 'nan2D', witch is in a DLL. So the two main modules are:
  • the engine (nan2d.dll)
  • The game executable (xyzzy.exe) all the rest of junk are game assets (art files, sound files, ...) that have no relationship with my problem. Now before I tell you about my problem, I want you to know that I have been struggling with this particular problem for exactly 8 days now. So before I say any more, I want to tell you that I might ‘pop’ at any minute and start talking out of my ass. Also, I really do hope that ‘we’ (as in all participants I this thread) can get rid of my problem… yes that would be very nice of you if you did have a hand in this. I cant stress enough how grateful I will be if someone helps me! So please bare with me and read about my problem. I am assuming that you know how a DLL works and what it can and can not do, here are some things about it:
  • A DLL is always loaded in memory, no matter how many applications use it… there is always one instance of it. Yeh that was only one… and I don’t see how that has to do anything with my problem. I am starting no type nonsense, lets get down to the problem shall we. In my engine, I have functions that take arguments (well duhh), and the engine function can read/write the contents of the variables (in the right scenario). In other word, my engine DLL works just the same as it would as it was statically linked as add-on code modules in my game project. So say that I have a function
      
    void foo(bar& changeMe)
    {
    	/* do some stuff */
    	changeMe.baz = qwert(); // whatever
    
    }  
    So that would be in my engine DLL. And of-cores I would have the correct import/export (ing) declarations before any functions)… I also use the __stdcall calling conversion. Okay, all that will be in my engine. Now in my game code:
      
    bar Somshit;
    foo(Someshit); /* whatever it doesn’t have to make any sense */  
    As you can see I am calling a function from my engine (the function would be declared in a header file) The memory for “Somshit” is allocated in the application heap (if I am pondering right), and an address is sent to the function when I call it (if I did not send the address way, with the &, then the “Somshit” variable from the application space would be copied to the DLL application space just for the function context, then destroyed when the context ends (begin={, end=})). Now, I am pretty sure that that will work without any fixing and whatnot. My application is larger… much larger… 20,000 lines of code and about 60% of that code calls engine functions (the engine itself is about 20,000 lines or less also) with may or may-not modify any variables. And I am sure that at least 20% of that engine calling code uses the engines INI files parser to get variables form INI files… witch require that the DLL to send data to the calling application heap. The problem that I am having is that the application is doing some vary strange shit at runtime. An example of it would be: I run the application in the debugger… nothing goes wrong, works like charm, 400+ FPS and all is smooth. Wait though, I am running in release mode with the debugger enabled, so I am not really debugging, I am just ruining in release mode (where no debugging information is stored in the application executables or any other debugging database that the compiler might generate when compiling all the code) AND having the debugger active... with is nothing more then a small spy (but whatever). Then I go to my desktop and click on the shortcut (witch launches the application form the BIN directory… the directory that it compiles to)… it starts… the “loading” sprite is displayed by the engine (meaning that the engine has successfully started)… a few seconds passes (witch only means that about 120 megabytes of data has already been parsed and read and whatever… but it means that the application is functioning normally, so far) then BAMO! I get an error stating “Unhandled exception at 0x77f51ec7 in XYZZY.exe: 0xC0000005: Access violation reading location 0x011bb77d.”, so I break it. I go to the most recognizable place in the stack, and I go to this block of code:
      
    LPTSTR strBase = pCfg->GetString( "Explosions","Explosion1SpriteBase" );
        LPTSTR strExt = pCfg->GetString( "Explosions","Explosion1SpriteExt" );
        N_NEWOBJ( pExplosion1,NAN2DAnimSprite() );
        for ( UINT i = 0; i < 30; i++ ) {
            TCHAR buf[MAX_PATH];
            PadName( strBase,strExt,buf,i );
            pExplosion1->AddSprite( buf,0 );
        }
        //_asm INT 3
    
        pExplosion1->BuildDefaultAnim();
        NAN2DSPRITE_ANIMATION Anim1 = pExplosion1->GetCurrentAnim();
        NAN2DSPRITE_ANIMATION Anim2 = pExplosion1->GetCurrentAnim();
        Anim2.Reverse();
        Anim1.AppendBack( Anim2.GetArray() );
        >>> ERROR <<< pExplosion1->SetAnim( Anim1 );
        pExplosion1->SetFPS( pCfg->GetFloat( "Explosions","Explosion1SpriteFPS" ) );
        pExplosion1->SetRotationCenterNonliner( 0.5f*fGRW,0.5f*fGRH );  
    What it does is just load some sprites from the information in an INI file… created a default animation (means it just sequences all the sprites in chronological order and makes an animation out of them)… get the animation that it just made 2 times. One of the animations it got is reversed, or flipped, so that it can be appended to the other clone to make a “bounce” animation (just Ping-Pong’s the animations). Then it sets the ready animation as the current animation in the pExplosion1 (type NAN2DAnimSprite) , witch goes to this code:
      
    /* Created: 2002/05/04 (Sat, May 04) 22:22 */
    INT NANCC NAN2DAnimSprite::SetAnim( NAN2DSPRITE_ANIMATION &TheAnim )
    {
        /* [5/4/2002] See if its already in the list */
        >>> ERROR <<< t_2DSpriteAnimArray::iterator result = find(Anim.begin(), Anim.end(), TheAnim);
        
        if (!(result == Anim.end())) {/* [5/4/2002] its in there! */
            SetAnimByID((*result).dwID);
            return ((*result).dwID);
        }else {
            return SetAnimByID(LoadAnim(TheAnim));
        }
        return -1;
    }  
    witch goes to this code:
      
    // TEMPLATE FUNCTION find
    
    template<class _InIt,
    	class _Ty> inline
    	_InIt find(_InIt _First, _InIt _Last, const _Ty& _Val)
    	{	// find first matching _Val
    
    	for (; _First != _Last; ++_First)
    		if (*_First == _Val)
    			break;
    	return (_First);
    	}  
    Then it just goes down the cryptic stack of STL function calls… Scary shit, I know. “So now you know” what I had to go through in the past 8 days . I joke you not, this is making me slowly go insane! In conclusion: I am not asking for you to tell me how to fix my code and just, I rather would like you to point me to a good resource on DLL programming and some place where I can find out EVERYTHING I need to know about making a DLL, the things that I can and can not do. I also really need to know what is the heap and how to use it wisely, please someone point me in the right direction. One more thing, I like to fix my own problems and dig myself out of my own grave so that is why I am asking you not to fix my code, but to point me in the right direction… but if you know your stuff about all this (memory, heap allocation, DLL) and consider yourself worthy of helping me fix this problem, then please tell me and I will strip the data files (required to get to the point in execution where the error occurs) to almost nothing (like 2x2x8 sprites), and send you the files required to compile everything in MSVS .NET (7.zero) so you can go over use your knowledge in all this to fix or help me fix the problem. I AM using Microsoft Visual Studio .NET! o_0 And most importantly, love yourself and you family… enjoy black history month [edited by - aftermath on January 24, 2003 7:19:33 PM]

    Share this post


    Link to post
    Share on other sites
    Advertisement
    quote:
    Original post by AfTeRmAtH
    this is sort of an advanced topic, so if you don’t know what a heap is


    heap is covered in compsci 2.
    quote:

    or what cross-boundary memory sharing is, you probably won’t follow me on my dumb explanation.


    i must've missed "cross-boundary" part of this post. anyway...
    quote:

    A DLL is always loaded in memory, no matter how many applications use it… there is always one instance of it.


    that was the case in win16. in win32, all dlls share their code segments, but unless you make the data segments shareable, they will be private to processes that uses the dll. under win9x, this would be accomplished by duplicating data segments with readwrite protection for every process, and under nt, data segments will have copy-on-write protection.
    quote:

    The memory for “Somshit” is allocated in the application heap (if I am pondering right), and an address is sent to the function when I call it (if I did not send the address way, with the &, then the “Somshit” variable from the application space would be copied to the DLL application space just for the function context, then destroyed when the context ends (begin={, end=})).


    there is no such thing as "dll address space". dll is using the address space of whatever process it is loaded in. in general, process objects (not threads, not modules, etc.) own most resources.
    quote:

    which requires that the DLL send data to the calling application heap.


    if you are allocating memory is dll with new/malloc and freeing it in exe with delete/free, you need to watch out for your c runtime settings. by default, exe and dll each get their own private little singlethreaded statically linked c runtime. this means that pointer allocated with new or malloc1 in one of them doesn't make any sense to the other2, as far as crt is concerned. it's a valid memory block in that you can read and write to it, but the other c runtime didn't allocate it, and thus doesn't know how to free it. the possible solutions are:
    - use multithreaded dll runtime for all modules in your project. this way there is only one copy of the c runtime for the entire process, and it will know how to handle everything from all modules.
    - provide resource cleanup functions in the dll that free memory that's allocated by the dll. in other words, if dll allocated a memory block, pass it to the dll to free it.

    1 this also holds true for other crt objects, such as file handles, FILE * pointers, and so on.
    2 if new/malloc are thin wrappers around HeapAlloc and don't add any of their data to memory allocated by heapalloc, as seems to be the case for retail c runtime, then theoretically pointer allocated by one c runtime should be freeable by any other c runtime, as there is only one win32 heap for your process (in contrast to potentially many crt "heaps").
    quote:

    The problem that I am having is that the application is doing some vary strange shit at runtime.
    An example of it would be:



    use watch/variables window to snoop around your data, and check if anything is corrupt. there isn't much to go on from the information you've provided. if your program is crashing while attempting to access supposedly perfectly valid data, you may have trashes your stack and/or memory earlier. global data breakpoints are helpful, but these are some of the nastier crashes you can get.

    you can also have debug information in release builds; just enable the corresponding options in c++ and linker settings. this will not make your program any slower or bigger, as debug info is stored separately in pdb files, although debugging optimized code can be fun.
    quote:

    I joke you not, this is making me slowly go insane!


    i believe you.
    quote:

    I rather would like you to point me to a good resource on DLL programming and some place where I can find out EVERYTHING I need to know about making a DLL, the things that I can and can not do.


    msdn and google have always worked for me, along with Jeffrey Richter's "advanced windows", which is now called "programming applications." excellent book if you want to know win32 up close and personal.
    quote:

    I also really need to know what is the heap and how to use it wisely, please someone point me in the right direction.


    heap is merely a piece of memory from which you can grab small chunks with functions like malloc. a big issue with heaps is heap fragmentation, which is when you get many small free blocks throughout the heap, but few or none large ones. in this scenario, successively allocating memory does not give you memory blocks that are close together physically, which kills the cache(s) on your processor. if your heap gets fragmented enough, you won't be able to allocate big objects even though there is enough combined free space in it for such allocations.

    one way of dealing with fragmentation problems is to have several heaps, each for a certain type(s) of objects. for instance, you can do all small allocations using new and delete, but if you need large memory blocks, let's say several kb in size, you allocate them from another heap (created with HeapCreate, allocate with HeapAlloc). if you have even bigger requests, ranging in hundreds of kilobytes or in megabytes, it might be better to use virtual memory instead (VirtualAlloc). virtual memory allocation functions don't use a specific block of memory that they have to grow every now and then for the allocations; instead, they give you memory straight from your address space, eliminating overhead involved in maintaining the heap. the downside is that VirtualAlloc rounds requests up to the processor page size (4 kb for x86), so using VirtualAlloc for allocations smaller than dozens of kilobytes is impractical.
    quote:

    One more thing, I like to fix my own problems and dig myself out of my own grave so that is why I am asking you not to fix my code, but to point me in the right direction…


    - watch window
    - examine any this pointers you have floating around
    - data breakpoints
    - debug info in release build
    - #define DEBUG even for release build, and see what happens
    - does this problem occur in debug builds? these have much more error checking thrown in
    quote:

    but if you know your stuff about all this ... I will strip the data files ... to almost nothing ... and send you the files required to compile everything in MSVS .NET ... so you can go over use your knowledge in all this to fix or help me fix the problem.


    mail the stuff my way, i'll look at it.

    [edited by - niyaw on January 21, 2003 6:54:00 AM]

    Share this post


    Link to post
    Share on other sites
    quote:

    A DLL is always loaded in memory, no matter how many applications use it… there is always one instance of it.


    This is not strictly true. You can compile dll''s to run in (at least) two different flavors. The first flavor recieves it''s own data segement everytime it''s loaded (I believe this is the VC default). It can also be compiled to share the same data segement with each instance.

    The few dll''s I use that rely on this behavior are written in Delphi, not VC, so I don''t know off-hand what the switches/settings are that affect this. Ought to be in the project settings, or perhaps in the .def file for the dll.


    A dll is essentially an .exe that is loaded into the same memory space as a prior exe. The Dll application space is the same as the application that invokes, as far as the data is concerned.
    However, the CRT (C run-time) memory allocators create seperate heaps for each one. You are not suppose to alloc in a dll and free in the app nor vice-versa (same for new & delete).

    If your dll creates an object, the dll needs a way to destroy it. This is one reason why COM uses reference-counting, and calling ->Release() on the object invokes delete (when the reference count falls to zero). This way the code that does the delete is compiled into the dll, not into the application.


    BTW, this is not really a software engineering question, it''s more about a specific debugging or dll issue. However, the ultimate solution to this problem might be a design issue, so I''ll let it slide this time

    Share this post


    Link to post
    Share on other sites
    Forum reply.

    Wow, thank you for the replies guys, it really means a lot to me. There is a lot to be gone over, so I am just going to refer to the first 2 posts then

    To Niyaw:
    I enjoyed reading your reply, it contained a lot of yummy brain food.
    quote:
    heap is covered in compsci 2.


    Lucky for me, I am a freshman high-skool student . I don’t know, you probably meant in collage right? The only SompScienc I have and will be having this year are
  • Intro to web design (pleh http://www.nan2D.com )
  • Computer graphics ( look above)
  • Architecture and 3D modeling.

    So nothing that has to do with programming, well maybe the intro to web-design thing because we will use HTML (witch is not a programming language by the way). Now back to topic I go!
    quote:

    I must've missed "cross-boundary" part of this post. anyway…

    No a fence, but I have no idea of what you mean by that, please explain
    quote:
    that was the case in win16. in win32, all DLLs share their code segments…

    So does that mean that say… here is the memory:
    .............
    DLL Space....
    .............
    ..(pr0n).....
    .............
    Application space.
    .............
    .............

    Does that mean that every time I invoke a function from “DLL Space”, it is always from the same space in memory, no matter what function calls it (I'm guessing that is wrong because then applications would tangle the DLL and make it do unexpected things because there a re to many asynchronous operations at one time); or does every “Application Space” have its local copy of the “DLL Space” (that seams more correct to me)… witch one is right?

    quote:
    there is no such thing as "dll address space". dll is using the address space of whatever process it is loaded in. in general, process objects (not threads, not modules, etc.) own most resources.


    Let me get this right, so like when I have some init. Code in a DLL, and it allocates some objects into memory, its actually just allocating those objects into the application space (identical as if those objects were being allocated by the code in the executable EXE)… am I understanding this correctly?

    Also, a CRT (C run-time) is basically a code module (shitload of functions that almost every application uses) this is present when an application is executed. It can be either a DLL or statically linked (meaning that the code from the .lib (in the “lib” directory of the compiler) will be included in my applications .obj files).
    Now there are two types of these modules:
  • A single threaded module that uses the parent processes (application, DLL, whatever…) memory, and does not access or share any outside memory. The memory that it allocated and de-allocates is only valid within the CRT’s process.
  • A multi-threaded module… this one I don’t get a lot about and how it works. I am guessing that it is a CRT module that it present to the calling process and all of its child processes (like the DLL’s that get loaded at run-time (not the ones loaded through LoadLibrary)). Now these things can happen and will be valid:

    o The parent process can create (allocate) memory in its heap via “new” or “malloc()”, or whatever… then those functions are invoked to the CRT witch places the newly allocated memory in the CRT heap (witch is “__crtheap” from looking at the source files), witch was created with, assuming, “CreateHeap()”.

    o Any child processes (a DLL for example) can allocate memory and it will be taken care of the same way as described above, placed in the same CRT heap.

    It looks like I am giving a lecture on the CRT and its types, I am not… I mealy want you (the guys that know this stuff) to tell me what I am getting wrong (and I think that I am getting a lot wrong). Thanks for reading this far too, it know its quite a bit of text, bare with me though.
    quote:
    global data breakpoints are helpful
    are those the errors when you don’t get a fancy shanty dialog box, instead a general MB_OK box telling you of an access violation? If not, what are you talking about?

    quote:
    you can also have debug information in release builds; just enable the corresponding options in c++ and linker settings. this will not make your program any slower or bigger, as debug info is stored separately in pdb files, although debugging optimized code can be fun.
    I am having these sort of crashes in both debug and retail builds.
    quote:
    I believe you.
    so far so good.

    quote:
    mail the stuff my way, I’ll look at it.
    if I just cant find any solution, I will send it to you (gota have MS .net though and DirectX 8.1)


    Now to Moderator Magmai :
    quote:
    This is not strictly true. You can compile dll's to run in (at least) two different flavors. The first flavor receives it's own data segments every time it's loaded (I believe this is the VC default). It can also be compiled to share the same data segement with each instance.

    Yes I know about that. It can be accomplished via pragmas and some directive where there is “.segment” and “.data” and whatever… that is not what I want though.

    Thanks for the help guys!

    Now for more questions (yes I know this Is a bit lengthy ).
    What does it mean when my heap is corrupted, and how may I have cause it.
    When I compile with the multithreaded debug versions of the CRT, I get the good’ol ASSERT(_Crt…); memory heap checking bs. What might be the problem… the MSDN docs just say that the function validates the heap and crap… but I don’t see what I might be doing that is corrupting it. With the retail versions of the CRT, everything works (well… prior to this 9 day problem that I am having ).

    What is the difference (and does it affect my code execution, like does this cause an error if I have it wrong) between a __cdecl function declaration and a __stdcall declaration. I remember having a __fastcall as my default declaration, and that worked too.

    And one more thing

    When I trace my code, the deadliest part of the code is when the CRT calls AllocHeap (or something like that), and one of its parameters is “__crtheap” meaning that it is allocating some memory onto the heap. Why would this cause errors if the Application and the engine DLL are both using the same CRT heap?

    Once again, thank you all for the help!



    [edited by - aftermath on January 21, 2003 11:50:55 PM]

    Share this post


    Link to post
    Share on other sites
    quote:
    Original post by AfTeRmAtH
    Lucky for me, I am a freshman high-skool student .


    in high school, the simple new/delete stuff is commonly found in the second semester of AP computer science AB course (www.collegeboard.org). computer science 2 is the college course where i'd expect to find equivalent issues discussed. now, obviously neither of them deals with dynamic-link libraries and the kind of problems you're experiencing.
    quote:

    No a fence, but I have no idea of what you mean by that, please explain


    well, cross-boundary means there has to be a boundary to cross. usually it's process boundary (exchanging data between different running processes, as in two .exe files) or computer boundary (exchange of data over network), or similar; in your case, all operations proceed in one address space, the address space of the process in question (more on this later), and there are no crossed boundaries.
    quote:

    So does that mean that say… here is the memory:
    .............
    DLL Space....
    .............
    ..(pr0n).....
    .............
    Application space.
    .............
    .............


    there's a distinction between virtual memory and physical memory . virtual memory is the kind of memory you're used to. you get virtual memory when you call malloc or new; when you take address of a variable, you get the address in the virtual address space. every running process has its own, private virtual address space . the size of the address space is 4 GB. when an exe is loaded, it looks like this (dots represent unused regions):

    .............
    .............
    .............
    exe code and data
    .............
    .............

    now some dlls are loaded, and the address space looks like this:

    .............
    .............
    kernel32.dll code and data
    .............
    .............
    user32.dll code and data
    .............
    .............
    yourdll.dll code and data
    someotherdll.dll code and data
    .............
    exe code and data
    .............

    note that everything belonging to each dll is stored together in the virtual address space. now let's say you make some heap allocations. the address space will now look like this:

    .............
    .............
    kernel32.dll
    .............
    default process heap allocated by kernel
    storage for object allocated with 'new int'
    free heap block
    storage for object allocated with 'new CoolClass'
    storage for object allocated with 'new char[10000]'
    more free heap blocks
    .............
    .............
    game.exe
    .............

    the important thing to note is that the process object, which is close, but not quite "the exe file", owns the address space. dlls don't own anything, they are merely pieces of memory within the address space that have a name.

    now, the virtual memory is not the memory you can find inside your computer. the stuff that's in ram chips is the physical memory . typically, size of your total physical memory (128, 256, 512, ... mb) is much less than combined size virtual address spaces of all running processes (4 gb * 30, 40, ...). how can this be? first, not all blocks of virtual address space are mapped to physical memory (ie, free blocks aren't; they are just marked free, and the operating system can use them, but they aren't occupying any physical memory). two, multiple pieces of virtual memory can reference the same piece of physical memory.

    consider a dll called cool.dll that only has code (no data). thus, there are no writes ever to the piece of memory that this dll occupies. now, suppose it is loaded in process A and process B. the layout of process A might be as follows:

    .............
    some kernel data
    .............
    cool.dll code
    .............
    some other data
    .............
    more data
    .............
    processA.exe code and data
    .............

    and layout of process B might be as follows:

    .............
    some other kernel data
    .............
    more kernel data
    .............
    someotherdll.dll code and data
    .............
    cool.dll code
    .............
    processB.exe code and data
    .............

    now, storing two read-only copies of code of cool.dll in your ram is clearly a waste of memory. thus, both cool.dll blocks in the two processes reference the same block of physical memory, which actually contains bits of code of cool.dll (in reality, contiguous virtual memory blocks don't have to correspond to contiguous physical memory blocks, and the way things work is more complicated). thus each process has its own copy of cool.dll, or at least each thinks that way, possibly loaded at different addresses, but there is only one instance of the bits of dll in your physical memory.

    Does that mean that every time I invoke a function from “DLL Space”, it is always from the same space in memory, no matter what function calls it

    pretty much, which is why dlls are so nice. there are 30 processes in your system, and each of them is referencing kernel32.dll that's 1 mb in size. 29 mb of waste are eliminated with this model.
    quote:

    (I'm guessing that is wrong because then applications would tangle the DLL and make it do unexpected things because there a re to many asynchronous operations at one time)


    not really, because the code is only readable. you can read the same location as much and as often as you want, no harm in that. things get tricky when you want to read and write to memory. which is where we get to data in dlls.
    quote:

    or does every “Application Space” have its local copy of the “DLL Space” (that seams more correct to me)…


    again, there is no such thing as "dll space". dll simply occupies a block of memory that is specially marked so that you know what it is; this block is part of the application address space.

    now, consider a dll called win.dll that has both code and data. the code writes to the data that's stored in a dll. the above pictures for virtual memory layouts for cool.dll are exactly the same as virtual memory layouts for win.dll, but the way the dll is stored physically is different. the code segment is still shared among all instances as described above; however, for the data segment, we have two scenarions.

    1: the data segment is explicitly marked as "shared". in this case, it's shared in exactly the same way as the code segment; different applications have the dll loaded at potentially different address, but they all reference the same physical piece of memory with dll data. if one application changes the data, other applications will read the changed data. if two applications try to change the data simultaneously, data will most likely be corrupted. this is where thread synchronization steps in, but i'm not going there now.

    2: data segment isn't marked as shared; this is the default. now each process thinks it has its own private copy of dll *data*, not just code. if process A modifies a variable in the dll data segment, process B expects to see the old value of the variable. how do we handle this?

    win 9x: for each process, a separate copy of the dll data segment is maintained in the *physical memory*. even if most of it is never changed, the data segment is still duplicated. effectively, each process now references a common physical memory block with dll code and a private physical memory block with dll data. this method is simple to implement, but wasteful.

    win nt: initially, the entire dll data is stored in a single physical memory block, like the code is. however, once a process tries to write to this block, a copy of the (small) part of it to which the process writes is made specifically for that process. any changes will therefore only be seen by the process which made them. after the write, the writing process will reference (most) of dll data from the shared physical memory block, but it will reference the modified (small) part of dll data from its copied physical memory block. this approach is more memory-efficient, but requires more clock cycles.

    i hope this wasn't too confusing; feel free to ask for clarification.
    quote:

    Let me get this right, so like when I have some init. Code in a DLL, and it allocates some objects into memory, its actually just allocating those objects into the application space (identical as if those objects were being allocated by the code in the executable EXE)… am I understanding this correctly?


    mostly. the objects are indeed allocated into the application space. however, problem with new/malloc is that they can attach their own management data to the memory block before giving them to you. that data may be used, for instance, to track memory leaks, or to measure memory load. it's those management blocks that are different from one c runtime to the other. note that you as a user never deal with them, they are internal to new/malloc implementation. again, feel free to ask for clarification.
    quote:

    Also, a CRT (C run-time) is basically a code module (shitload of functions that almost every application uses) this is present when an application is executed. It can be either a DLL or statically linked (meaning that the code from the .lib (in the “lib” directory of the compiler) will be included in my applications .obj files).


    yes. note that eventually the crt code will be in your application address space, be that through a crt dll or through part of your exe file in case of static linking.
    quote:

    Now there are two types of these modules:
  • A single threaded module that uses the parent processes (application, DLL, whatever…) memory, and does not access or share any outside memory. The memory that it allocated and de-allocates is only valid within the CRT’s process.


  • almost. the crt doesn't use its parent process for anything. what happens is that this particular instance of crt may add some internal variables to the allocated memory block, and other crts *in the same process* won't make sense of them. keep in mind that we're talking about things occuring entirely within a single process (application).
    quote:

  • A multi-threaded module… this one I don’t get a lot about and how it works.


  • multithreaded static crt functions just like the singlethreaded, but it provides thread synchronization so that if two threads try to access the same data at the same time (ie, ask for some memory to be allocated at the same time) one of them is paused while the other gets its request satisfied. this avoids the problem of two threads modifying the same data simultaneously or one of them reading data that the other one is modifying that i mentioned earlier.

    the really different crt is multithreaded *dll* one. with it (provided you use this setting for all dlls and your exe, not just for some) there effectively is only one runtime library for the entire process, in contrast to many crts you'll get if you have dlls linked against any of the two static libraries. while the crt still uses some internal management variables to keep track of its activities, *all modules* in your application reference the same runtime, and it can make sense of the variables that it itself allocated. notice how this is different from static-linked crt scenario, where one crt is asked to make sense of internal variables initialized by another crt *running in the same process*.
    quote:

    I am guessing that it is a CRT module that it present to the calling process and all of its child processes (like the DLL’s that get loaded at run-time (not the ones loaded through LoadLibrary)).


    you seem to be confusing processes and modules (dll modules in particular). stay within one process. there are no child processes around; a child process means you execute another exe file and thus create a new process with a new address space with a new dll set loaded, etc. it has very little to do with loading another dll via LoadLibrary.
    quote:

    Now these things can happen and will be valid:

    o The parent process can create (allocate) memory in its heap via “new” or “malloc()”, or whatever… then those functions are invoked to the CRT witch places the newly allocated memory in the CRT heap (witch is “__crtheap” from looking at the source files), witch was created with, assuming, “CreateHeap()”.

    o Any child processes (a DLL for example) can allocate memory and it will be taken care of the same way as described above, placed in the same CRT heap.


    this doesn't make much sense. again, child process and a dll have very little in common. stay within one process.

    a word on __crtheap: that is the handle to the *application* win32 heap. there is one default heap for every process, and ultimately all memory allocations made with new/alloc are made from it (simplification; actually, this does not always have to be the case, but it's fine for our discussion). again, note that allocating memory with new/malloc from exe and dll:
    - gives you memory from the same default win32 process heap (you may even get the same pointer)
    - gives you memory that *anything* in your process has read and write access to. this includes code in your exe, code in your dlls, and code in any other dlls that might be present, such as windows dlls.
    - differs only by the internal management data that c runtime inserts into the allocated memory.
    quote:

    It looks like I am giving a lecture on the CRT and its types, I am not…


    i seem to be giving one
    quote:

    are those the errors when you don’t get a fancy shanty dialog box, instead a general MB_OK box telling you of an access violation? If not, what are you talking about?


    ctrl+b, data tab. i'm not sure if you can enter a variable name in; i stick in a pointer address that i want to watch casted to some pointer type. let's say i have a variable at address 0x1234 that is trashed by something; i would set a breakpoint at (int*) 0x1234 and wait for it to be hit, then look around call stack/watch windows to see what hit it. of course, you must know the address of the variable to look at, and you must initialize the breakpoint when the value in it is still good. note that restarting your program will usually invalidate all hardcoded memory breakpoints (that is, they will point to some different memory area, and you'll have to reenter new memory addresses).
    quote:

    What does it mean when my heap is corrupted, and how may I have cause it.


    it usually means that the supporting data structures of the heap are overwritten with junk. let's say that heap maintains a linked list of free blocks; if you've trashed the 'next' pointer in one of the nodes by writing beyond bounds of a new-allocated array, the heap manager will not be able to go through all nodes and will say that heap is corrupt.
    quote:

    When I compile with the multithreaded debug versions of the CRT, I get the good’ol ASSERT(_Crt…); memory heap checking bs. What might be the problem…


    out of bounds array access, freeing memory twice, writing to a freed memory block, writing to a not-yet-allocated memory block (this shouldn't be the case usually), among the most common. post exact error message and whatever else the debugger is telling you. look at the call stack and variables window, as usual, to see which variables are bad. look for magic values, such as 0xcc, 0xfeee, 0xcdcd, and similar.
    quote:

    With the retail versions of the CRT, everything works (well… prior to this 9 day problem that I am having ).


    always fix your problems in debug builds. building in release merely delays the problem until its detection is between hard and impossible, which is what you've experienced based on your original post. remember: bugs very rarely go away by themselves. get a clean run in debug build first, until you do so, forget about building in release.
    quote:

    What is the difference (and does it affect my code execution, like does this cause an error if I have it wrong) between a __cdecl function declaration and a __stdcall declaration. I remember having a __fastcall as my default declaration, and that worked too.


    cdecl is the only calling convention out of the two supporting variable number of arguments to the calling function. stdcall's cleanup code is slightly faster when compared with cdecl on a function-by-function basis. however, these days msvc doesn't adjust esp after every function call, which means that cdecl may end up being faster than stdcall. i'd keep the default calling convention on cdecl, and make sure you always match calling conventions when you cast function pointers. calling convention mismatches are known to cause unrelated crashes and/or memory corruptions elsewhere in your code.
    quote:

    When I trace my code, the deadliest part of the code is when the CRT calls AllocHeap (or something like that), and one of its parameters is “__crtheap” meaning that it is allocating some memory onto the heap. Why would this cause errors if the Application and the engine DLL are both using the same CRT heap?


    see above; it's because of the internal crt structures.

    feel free to ask more.

    do you get the feeling that these things take a while to type?

    [edited by - niyaw on January 22, 2003 1:08:11 AM]

    Share this post


    Link to post
    Share on other sites
    The only time I''ve had to explicitly deal with cross-process memory sharing is when I use a WindowsHook (for mouse/keyboard hooking for all applications), since my .dll gets loaded into each application, and grabs only the keys for the app it''s loaded into.

    Since your .dll only has to talk to the .exe it''s loaded in, you don''t need to do anything funky to get it to work. I suspect that your bug may lie somewhere else in your system.

    Share this post


    Link to post
    Share on other sites
    Guest Anonymous Poster
    One thing that''s been overlooked in the original post is that problems arise when the code is switched from a debug build to a release build.

    AFAIK, with MSVC, in a debug build, local variables are initialised to zero whereas in a release build, the local variables are not initialised and contain the values that are present in the memory when they are assigned addresses (which will invariably won''t be zero), if you see what I mean.

    So, for example:

    void afunc (void)
    {
    int a;
    if (a == 0) // compiles in MSVC V6.0 without warnings in release mode!
    {
    // true in a debug build
    }
    else
    {
    // 255 out of 256 times will be true in release
    }
    }

    So, either initialise everything or create your own assert style macro that can be compiled into the code for both relases and debug builds and then add lots of asserts into your code. It''s good engineering practice to validate all parameters prior to using them (to make a reference to the forum name).

    Skizz

    Share this post


    Link to post
    Share on other sites
    quote:
    Original post by Anonymous Poster
    One thing that's been overlooked in the original post is that problems arise when the code is switched from a debug build to a release build.


    not quite. debug build was already complaining about broken heap.
    quote:

    AFAIK, with MSVC, in a debug build, local variables are initialised to zero


    not quite. they are initialized to 0xCCCCCCCC.
    quote:

    whereas in a release build, the local variables are not initialised and contain the values that are present in the memory when they are assigned addresses (which will invariably won't be zero), if you see what I mean.


    at least you got this right.
    quote:

    So, for example:

    void afunc (void)
    {
    int a;
    if (a == 0) // compiles in MSVC V6.0 without warnings in release mode!


    set your warning level to 4, and you'll get one. learn to use your compiler before complaining about its shortcomings.
    quote:

    {
    // true in a debug build
    }


    msvc6: false in debug build, due to the reason i stated above.

    msvc7, which original poster is using: not only false, but that line will cause a runtime error, "accessing an uninitialized variable", if you use the runtime checks provided by msvc. these checks should be enabled by default in debug builds.
    quote:

    else
    {
    // 255 out of 256 times will be true in release
    }
    }


    i love %d out of %d and %.3lf-formatted statistical figures. would you mind sharing your source?


    [edited by - niyaw on January 23, 2003 4:34:08 PM]

    Share this post


    Link to post
    Share on other sites
    Guest Anonymous Poster
    Ok, I''ve been programming Java for far too long now. :-( And I''ve not the £1000+ spare to migrate to V7 (plus upgrading my PC to run it, £100''s more), the cost / benefit just isn''t viable.

    Anyway, my example wasn''t the best I admit.

    But, in my defence, the original post did say:

    "I run the application in the debugger… nothing goes wrong, works like charm, 400+ FPS and all is smooth. Wait though, I am running in release mode with the debugger enabled, [cut] then BAMO! I get an error stating “Unhandled exception at 0x77f51ec7 in XYZZY.exe: 0xC0000005: Access violation reading location 0x011bb77d.”, so I break it."

    There''s no mention of it going tits-up in a debug build.

    But you''re right, 0xcc and not 0! Doh!

    My example wasn''t the best either. Had I have put ''char a'' instead of ''int a'' then it would be a given value, statistically speaking, 1 out of 256 times. Sorry about the confusion.

    I have a thing about warning level 4 from a previous MS compiler that would throw up numerous (read: pages of) warnings about ''windows.h''. But, checking it now it compiles fine. Bizarre, MS fixing something ;-)

    Personally speaking, I understand my compiler very well (but, like I said above, it''s been a while and I must get re-acquainted soon). You''re obviously one of the very small minority that does too - I''ve been programming games professionally for 12 years and have yet to meet more than a couple of programmers that knows what the compiler is doing (especially the precompiled header options!). Although it can go too far, I''ve been to a presentation about this where the speaker was writing incredibly convoluted and obfuscated C code to get the compiler to generate the optimum ASM code he wanted - which was really bad engineering as the output was dependant not only on the compiler version but the compiler options used! He should have just written it in ASM to start with - it would have been much clearer.

    Anyway, it wasn''t the best of examples, but the concept is sound - data is initialised in a debug build, whereas in a release build it isn''t. It applies more to dynamically allocated memory (as the compiler can''t check that the data is initialised) than to local variables. (And in this case, in debug, it''s initialised to 0xcd).

    One other difference between the two builds that I''ve seen cause problems in the past is writing past either end of a dynamically allocated memory block. In debug build this won''t causes a problem, if it''s just a few bytes over, as there are guard blocks present (0xfd''s). In release, you''d be overwriting heap memory management data, thus corrupting the heap.

    Skizz

    Share this post


    Link to post
    Share on other sites
    quote:
    Original post by Anonymous Poster
    I have a thing about warning level 4 from a previous MS compiler that would throw up numerous (read: pages of) warnings about ''windows.h''.


    which is why you compile standard headers on level 3 with pragma warning push/pop, and your code on level 4. same goes for stl.
    quote:

    But, checking it now it compiles fine. Bizarre, MS fixing something ;-)


    yes indeed.
    quote:

    Anyway, it wasn''t the best of examples, but the concept is sound - data is initialised in a debug build, whereas in a release build it isn''t. It applies more to dynamically allocated memory (as the compiler can''t check that the data is initialised) than to local variables. (And in this case, in debug, it''s initialised to 0xcd).


    i think you''re (still) implying that debug builds hide bugs that show up in release builds. correct me if i''m wrong. but if i''m right, it''s plainly not so. debug capabilities of msvc are incredible for finding and fixing bugs. granted, they won''t do much for a programmer who "debugs" with print statements or log files, but those magic value often tell you exactly what the problem is: accessing uninitialized memory, writing past allocated memory, freeing memory twice, accessing freed memory. msvc''s debug heap detects underwrites and overwrites, double-frees, memory leaks, memory states and state differences, breaking on allocation id, and custom alloc hooks. debugging broken debug heap in msvc is so ridiculously easy that it''s not even funny, compared to let''s say tracking down broken kernel heap (exception in kernel32: bad heap pointer passed from shell32, that''s five functions away from your code or there are no your symbols in the call stack, now what?). msvc7''s runtime checks are even better: detection of buffer overflows (never had this occur on me yet, so can''t comment), catching uninitialized variables, and so on.

    of course, warning level 4 is mandatory. why spend time debugging when your compiler can tell you *exactly* what the problem is? level 4 catches assignments in if statemtents and reads of uninitialized data, among other things.
    quote:

    One other difference between the two builds that I''ve seen cause problems in the past is writing past either end of a dynamically allocated memory block. In debug build this won''t causes a problem, if it''s just a few bytes over, as there are guard blocks present (0xfd''s). In release, you''d be overwriting heap memory management data, thus corrupting the heap.


    again, in debug build you''ll get an assertion from debug heap telling you that you wrote past allocated memory. not only that, you''ll be given allocation number, on which you can break next time you run the program. one minute later, you know exactly which object or array writes past itself.

    there''s no way you can blame msvc debug options for hindering your ability to debug programs. it''s just the opposite.

    Share this post


    Link to post
    Share on other sites

    • Advertisement
    ×

    Important Information

    By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

    We are the game development community.

    Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

    Sign me up!