As far as I am aware, ripping components of a game engine into seperate dll libraries is a fairly common practice to allow for easy and simple upates ect...
But I am curious about the difficulty of implementing such a modular system. I don't have much expieriance doing something like this so I figured I should see if someone can give me some insight on this.
When working dll's, there are subtle things to worry about. So one question I have is how dll's communicate with other dll's and share data. There are issues of dll's having there individual heap, which could cause program crashes if you try to allocate, then deallocate in a different dll ect... But also, can't dlls share the heap with the loading exe app? There are issues with returning objects from dlls and crossing the dll boundries ect... I have read over some of these things that are vaugly described, and I am looking for some one to guide me in the general process of creating a modular system like this.
Thanks for any comments you all have!
Edit: Aswell as the different uses of linking(implicit or explicit[Load Library or link to the .lib]) These all are factors that can, and will go wrong.
Creating DLL's from your game engine
I started doing this. I kinda gave up after I started making entire separate dlls for every tiny little thing.
Instead I'm just going to have one giant library for the engine itself.
I was looking in the Crysis folders and saw they had a lot of separate dlls called things like CryThread.dll, CryFont.dll, so they managed to split up the different parts pretty nicely.
I also found that in theory a lot of systems can easily be separate, but there are still many dependancies between everything. It's definitely good to try to decouple systems as much as you can but it doesn't work out quite that well in theory.
For example the developer console is it's own system.
The graphics depend on the developer console for all the variables and console commands to do with the graphics system.
The developer console depends on the graphics to display itself.
Basically you start needing to make things extremely generic which takes a lot of effort. While before I could hook two systems up in five minutes, I now need to take an hour or so to think about how to create complex generic interfaces between them and then implement it. This would be fine if I wanted to release, let's say, a Developer Console library for the community to use in their games. I have no interest in that though, all I want to do is make a game rather than spending hours of my day making everything with extremely good design. So I do good design but not perfect design.
In the end I have faster results. An average person playing my game isn't going to think, "OH MY GOD, WHAT AMAZING DESIGN AND DECOUPLING THEY HAVE BETWEEN ALL THE COMPONENTS!!!!" They'll see an awesome game that I was able to make quickly.
On the other hand if you worry about absolutely perfect design at all times, you'll be stuck rewriting your engine constantly for years with no results.
It took me about 6 years of making games before I learned this, and I'm still trying to transition into that line of thinking. It's just hard because I care about good design a little too much sometimes.
So basicacally it's good to draw the line somewhere. The reailty is, it won't be all that much different to patch one dll or to just patch the entire 3-6MB executable that is your game. It might seem a lot cleaner but it takes a lot more effort and thought without very much benefit to split up an engine into different dlls.
In the end I decided to just have my engine be a giant statically linked library. If I'm making a tool for the engine that uses the renderer, the tool uses the renderer and no other part of the engine. The person making the tool still has access to the ENTIRE engine library which might seem a bit unclean but really it's no different than having access to the entire C standard library when all you want to do is make a "Hello World" application or something.
The important thing is results. Did you deliver the tool maker the necessary library functionality so he can make use of your renderer? Or are you spending months constantly rewriting your engine to have better encapsulation between the components while the person needing your renderer is left waiting?
Now imagine instead that that's a game you could have potentially released and gotten praise for from random gamers who couldn't care what so ever how you made that game.
(LOL in the end this rant became more something aimed at one of my friends...)
Instead I'm just going to have one giant library for the engine itself.
I was looking in the Crysis folders and saw they had a lot of separate dlls called things like CryThread.dll, CryFont.dll, so they managed to split up the different parts pretty nicely.
I also found that in theory a lot of systems can easily be separate, but there are still many dependancies between everything. It's definitely good to try to decouple systems as much as you can but it doesn't work out quite that well in theory.
For example the developer console is it's own system.
The graphics depend on the developer console for all the variables and console commands to do with the graphics system.
The developer console depends on the graphics to display itself.
Basically you start needing to make things extremely generic which takes a lot of effort. While before I could hook two systems up in five minutes, I now need to take an hour or so to think about how to create complex generic interfaces between them and then implement it. This would be fine if I wanted to release, let's say, a Developer Console library for the community to use in their games. I have no interest in that though, all I want to do is make a game rather than spending hours of my day making everything with extremely good design. So I do good design but not perfect design.
In the end I have faster results. An average person playing my game isn't going to think, "OH MY GOD, WHAT AMAZING DESIGN AND DECOUPLING THEY HAVE BETWEEN ALL THE COMPONENTS!!!!" They'll see an awesome game that I was able to make quickly.
On the other hand if you worry about absolutely perfect design at all times, you'll be stuck rewriting your engine constantly for years with no results.
It took me about 6 years of making games before I learned this, and I'm still trying to transition into that line of thinking. It's just hard because I care about good design a little too much sometimes.
So basicacally it's good to draw the line somewhere. The reailty is, it won't be all that much different to patch one dll or to just patch the entire 3-6MB executable that is your game. It might seem a lot cleaner but it takes a lot more effort and thought without very much benefit to split up an engine into different dlls.
In the end I decided to just have my engine be a giant statically linked library. If I'm making a tool for the engine that uses the renderer, the tool uses the renderer and no other part of the engine. The person making the tool still has access to the ENTIRE engine library which might seem a bit unclean but really it's no different than having access to the entire C standard library when all you want to do is make a "Hello World" application or something.
The important thing is results. Did you deliver the tool maker the necessary library functionality so he can make use of your renderer? Or are you spending months constantly rewriting your engine to have better encapsulation between the components while the person needing your renderer is left waiting?
Now imagine instead that that's a game you could have potentially released and gotten praise for from random gamers who couldn't care what so ever how you made that game.
(LOL in the end this rant became more something aimed at one of my friends...)
Decoupling of this nature is really only beneficial (IMHO) when you need to isolate the work of many different individual programmers; it's great for making sure that people's systems obey well-defined interfaces and contracts, but it has a huge cost overhead as well. The costs are primarily mental and architectural, but if you aren't insanely careful, you can create performance and memory weaknesses as well.
In my opinion, it's really only worth messing with this kind of decoupling if you absolutely need it. Otherwise, just get something cool done.
In my opinion, it's really only worth messing with this kind of decoupling if you absolutely need it. Otherwise, just get something cool done.
The biggest purpose for DLLs (or "shared libraries") is when you plan to share your library (go figure) among multiple applications/processes, not multiple developers. Don't confuse something designed to improve programmer efficiency (the use of OOD, well-defined interfaces, code reviews, etc.), with something that was originally designed to address distribution and support of multple programs that want to use the same shared code (VC runtimes are a great example). What is being discussed here is really two different beasties.
First Important Point: Whether you have 1 mammoth DLL or 270 tiny DLLs (like the app I am Lead Dev for does) has nothing to do with the fact that the right way to program large apps is to come up with solid, generic interfaces and code to them. This makes your code more modular, more reusable, and allows you to put more people on the project without breaking each other on a daily basis. This is one of the fundamentals of OOP, and has nothing to do with how you link your stuff. You isolate programmers by having them change encapsulated implementation details. No one cares if the code they are writing ends up in a static library, a dynamic library that gets linked into a DLL, or how many DLLs there are. What they WILL care about is that you had a call yesterday that took 3 parameters, and today it takes 4, and they have to change their code in 150 places. DLL's don't touch this issue at all, for good or bad. The way to enforce proper programming approaches is through solid interface design, and code reviews to make sure people are sticking to it.
Second Important Point: There are many, many, many problems with DLLs, and many complicated systems have been invented to try and resolve them (start your learning journey with the search phrases "Windows COM", and "DLL hell"), and the problems just keep getting more and more complex as the solutions do. Deciding whether you want a bunch of DLLs is really an easy ask: would you rather manage 4 files to distribute, or 400? DLL's serve a purpose, but most of those purposes remain down lower than you need to worry about. VC runtimes make great DLL's. OS code makes great DLL's. Things that a LOT of apps need access to, and need frequent updates - THESE make great DLLs.
Now, let's say you're building a game engine. Unless you plan to a) use that engine in several games; and b) want to be able to update the engine with application X and allow application Y to benefit from the improvements - then you don't need a DLL. In fact, it can cause a lot of problems if you TRY to share a DLL between two different apps like this, when they end up being incompatible (see "DLL hell", referenced above). In general, what really happens is you have version 1 of an engine that you ship with App X. Then when you make App Y, you improve the engine, and ship Engine version 2 with App Y, and so on. If you now try to run App X and it loads Engine 2, well, unless you're World-Class good at forward-design (also known as "psychic", "lucky", or "liar") - then you will have not anticipated something. Then you've either got multiple versions of the same interface all doing different things, or some other kludge - but all to the same effect: different engine code for different apps.
Current best practice is to create a small bootstrapper exe, and then gob as much stuff as makes sense into a single DLL. For a great example of this, see Google Chrome. They have a few DLLs, but most of their code ships in one (relatively) large chrome.dll.
I've come down pretty hard on DLLs, so I'll try to advocate for them at least a little. A great use for DLL's is if you are creating something that you want to sell to someone else, like an SDK. Then, you package your stuff up in a DLL, give them the header files and (hopefully) expanded documentation, and then they can use your code without having access to it, so you can retain the IP.
First Important Point: Whether you have 1 mammoth DLL or 270 tiny DLLs (like the app I am Lead Dev for does) has nothing to do with the fact that the right way to program large apps is to come up with solid, generic interfaces and code to them. This makes your code more modular, more reusable, and allows you to put more people on the project without breaking each other on a daily basis. This is one of the fundamentals of OOP, and has nothing to do with how you link your stuff. You isolate programmers by having them change encapsulated implementation details. No one cares if the code they are writing ends up in a static library, a dynamic library that gets linked into a DLL, or how many DLLs there are. What they WILL care about is that you had a call yesterday that took 3 parameters, and today it takes 4, and they have to change their code in 150 places. DLL's don't touch this issue at all, for good or bad. The way to enforce proper programming approaches is through solid interface design, and code reviews to make sure people are sticking to it.
Second Important Point: There are many, many, many problems with DLLs, and many complicated systems have been invented to try and resolve them (start your learning journey with the search phrases "Windows COM", and "DLL hell"), and the problems just keep getting more and more complex as the solutions do. Deciding whether you want a bunch of DLLs is really an easy ask: would you rather manage 4 files to distribute, or 400? DLL's serve a purpose, but most of those purposes remain down lower than you need to worry about. VC runtimes make great DLL's. OS code makes great DLL's. Things that a LOT of apps need access to, and need frequent updates - THESE make great DLLs.
Now, let's say you're building a game engine. Unless you plan to a) use that engine in several games; and b) want to be able to update the engine with application X and allow application Y to benefit from the improvements - then you don't need a DLL. In fact, it can cause a lot of problems if you TRY to share a DLL between two different apps like this, when they end up being incompatible (see "DLL hell", referenced above). In general, what really happens is you have version 1 of an engine that you ship with App X. Then when you make App Y, you improve the engine, and ship Engine version 2 with App Y, and so on. If you now try to run App X and it loads Engine 2, well, unless you're World-Class good at forward-design (also known as "psychic", "lucky", or "liar") - then you will have not anticipated something. Then you've either got multiple versions of the same interface all doing different things, or some other kludge - but all to the same effect: different engine code for different apps.
Current best practice is to create a small bootstrapper exe, and then gob as much stuff as makes sense into a single DLL. For a great example of this, see Google Chrome. They have a few DLLs, but most of their code ships in one (relatively) large chrome.dll.
I've come down pretty hard on DLLs, so I'll try to advocate for them at least a little. A great use for DLL's is if you are creating something that you want to sell to someone else, like an SDK. Then, you package your stuff up in a DLL, give them the header files and (hopefully) expanded documentation, and then they can use your code without having access to it, so you can retain the IP.
Also, sorry to spam, but if you're thinking about using DLLs anyway, and you need to pass stuff across DLLs or across processes, you can start here:
http://msdn.microsoft.com/en-us/library/h90dkhs0(v=vs.80).aspx
I had a case where I needed a static variable inside a class in a DLL (it was used to prevent dialog re-entrancy). Unfortunately, this DLL was being used by two different apps (a desktop app and also a browser plugin), so they were two totally separate memory spaces, so two different static variables. The only way to fix this was to use solutions from the above. Really kludgy. Hate it.
But again, this was because I had *multiple processes* that needed to run the same code at the same time, so it actually was the right solution (short of using some kind of OS-based semaphore). Most games (or apps in general) don't operate like this, and aren't well served by the additional complexity.
http://msdn.microsoft.com/en-us/library/h90dkhs0(v=vs.80).aspx
I had a case where I needed a static variable inside a class in a DLL (it was used to prevent dialog re-entrancy). Unfortunately, this DLL was being used by two different apps (a desktop app and also a browser plugin), so they were two totally separate memory spaces, so two different static variables. The only way to fix this was to use solutions from the above. Really kludgy. Hate it.
But again, this was because I had *multiple processes* that needed to run the same code at the same time, so it actually was the right solution (short of using some kind of OS-based semaphore). Most games (or apps in general) don't operate like this, and aren't well served by the additional complexity.
The biggest purpose for DLLs (or "shared libraries") is when you plan to share your library (go figure) among multiple applications/processes, not multiple developers.
Normally, it's done to improve build times on large projects (static libs can takes an age to link....).
Second Important Point: There are many, many, many problems with DLLs, and many complicated systems have been invented to try and resolve them (start your learning journey with the search phrases "Windows COM", and "DLL hell"), and the problems just keep getting more and more complex as the solutions do. Deciding whether you want a bunch of DLLs is really an easy ask: would you rather manage 4 files to distribute, or 400? DLL's serve a purpose, but most of those purposes remain down lower than you need to worry about. VC runtimes make great DLL's. OS code makes great DLL's. Things that a LOT of apps need access to, and need frequent updates - THESE make great DLLs.
When you say 'many problems' I assume you basically mean: STL. If you avoid using STL in your DLL headers (or any other class that changes size between release and debug), most of the issues are relatively trivial. Even then, this only really applies if you start mixing debug and release code (the debug and release CRTs can get into arguments). If you just replace a static lib with a DLL, and you properly export the functions (and data), then you won't have problems....
Now, let's say you're building a game engine. Unless you plan to a) use that engine in several games; and b) want to be able to update the engine with application X and allow application Y to benefit from the improvements - then you don't need a DLL. In fact, it can cause a lot of problems if you TRY to share a DLL between two different apps like this, when they end up being incompatible (see "DLL hell", referenced above). [/quote]
Nonsense. Set up your build rules properly so that doesn't happen.
In general, what really happens is you have version 1 of an engine that you ship with App X. Then when you make App Y, you improve the engine, and ship Engine version 2 with App Y, and so on. If you now try to run App X and it loads Engine 2, well, unless you're World-Class good at forward-design (also known as "psychic", "lucky", or "liar") - then you will have not anticipated something. Then you've either got multiple versions of the same interface all doing different things, or some other kludge - but all to the same effect: different engine code for different apps.
The days of writing a shit installer that blindly copies DLLs to system32 are long gone. If you are still doing this, then really you should maybe stop doing it? Most people would include the DLL in the installer, and it would be installed in the same dir as your exe. Doing anything else is completely mental.....
Current best practice is to create a small bootstrapper exe, and then gob as much stuff as makes sense into a single DLL. For a great example of this, see Google Chrome. They have a few DLLs, but most of their code ships in one (relatively) large chrome.dll.
That is not best practice, that's just mental. One would imagine that chrome.dll is actually a series of DLLs that has been combined into a single dll for release only (there are tools included with every compiler that do this!). If the devs of chrome had all of their code in a single dll, a single change would take hours to re-link!
I've come down pretty hard on DLLs, so I'll try to advocate for them at least a little. A great use for DLL's is if you are creating something that you want to sell to someone else, like an SDK. Then, you package your stuff up in a DLL, give them the header files and (hopefully) expanded documentation, and then they can use your code without having access to it, so you can retain the IP.
.... if you ever experience a problem with DLL's, it will be because you try to do ^^this^^. DLL hell begins and ends with plugin architectures (It's something I could talk about for hours, but I'll spare you the details....). Simply using a DLL instead of a static lib won't cause problems.
It seems we could get into a very heated religious debate on this topic, and I don't really see the point in simply rehashing all the written material already on the web, but I'll try to focus my response:
Simply using a DLL instead of a static lib won't cause problems.
[/quote]
Simply using a static lib instead of a DLL won't prevent bad programming practice, which seemed to be what the first 2 posters implied.
Your response, on the other hand, seems to revolve mainly around improving link times. This statement:
If the devs of chrome had all of their code in a single dll, a single change would take hours to re-link!
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]
[/quote][/font]
Seems to me to be just blatantly false.
The application I work on is significantly larger than Chrome (total size of our DLLs is 746 megs (debug), so guesstimate half for release. total size of "chrome.dll" is 28 megs). I can build my entire project from scratch, including compile and link time, in about 20 minutes. Why would you think it would take "hours" to re-link such a small DLL?
EDIT: [color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]Also, visual studio does incremental linking to speed up the process by default in debug builds anyway, so you're probably not re-linking the whole thing...
[/font]
Also, visual studio does incremental linking to speed up the process by default in debug builds anyway, so you're probably not re-linking the whole thing...
As this is a religious debate in many cases, I would direct the OP to do his own research. Here are some links to get started:
"Advantages of Using DLL's" - straight from the Horse's mouth:
http://msdn.microsoft.com/en-us/library/dtba4t8b.aspx
Read this page carefully and consider how many of the "advantages" actually are true in practice, or apply to you at all.
These may be not very in-depth, but if you read the comments they all make the same basic point. The only time it is beneficial to break code into deployment packages if you really intend to share that code among applications.
http://stackoverflow.com/questions/2881296/one-big-executable-or-many-small-dlls
http://stackoverflow.com/questions/1592644/which-is-better-one-dll-for-all-or-many-dlls
http://forums.asp.net/t/461506.aspx/1
http://www.codeproject.com/Questions/259523/Multiple-DLLs-Vs-Single-DLL
I got these by googleing "one dll vs many" or similar strings. I did not cherry pick, but you'll see in most cases the answer is the same. One DLL unless you have a good reason.
In the end, this decision, like all programming, comes down to trade-offs. Know what it is you're trying to do, evaluate the benefits and costs in light of your goals, and then make an educated guess. Don't spend too long on it, because it's holding up your development efforts. ;)
"Advantages of Using DLL's" - straight from the Horse's mouth:
http://msdn.microsoft.com/en-us/library/dtba4t8b.aspx
Read this page carefully and consider how many of the "advantages" actually are true in practice, or apply to you at all.
These may be not very in-depth, but if you read the comments they all make the same basic point. The only time it is beneficial to break code into deployment packages if you really intend to share that code among applications.
http://stackoverflow.com/questions/2881296/one-big-executable-or-many-small-dlls
http://stackoverflow.com/questions/1592644/which-is-better-one-dll-for-all-or-many-dlls
http://forums.asp.net/t/461506.aspx/1
http://www.codeproject.com/Questions/259523/Multiple-DLLs-Vs-Single-DLL
I got these by googleing "one dll vs many" or similar strings. I did not cherry pick, but you'll see in most cases the answer is the same. One DLL unless you have a good reason.
In the end, this decision, like all programming, comes down to trade-offs. Know what it is you're trying to do, evaluate the benefits and costs in light of your goals, and then make an educated guess. Don't spend too long on it, because it's holding up your development efforts. ;)
software.intel.com has a demo called ‘smoke’ where they showcase a modular design in conjunction with multithreaded coordination. I suggest to start there to get a impression how such design could be implemented.
However to come back on the op’s original question(s). In visual c++ or rather said Windows itself heaps are created and destroyed by respectively CreateHeap, DestroyHeap. In the prologue or epilogue of the respective entry or exit points of an application/dll such heap is created or destroyed. To my recollection the handle of the heap drives malloc/free and likewise new/delete calls. Visual Studio installations accommodates the source code of the crt (common runtime) and much of this behavior could be altered if really desired.
That being said. Sharing heaps should be technically possible. The wisest thing is to ignore it and attempt to solve the complications of data ownership with reason, architecture and flair. For instance you might be tempted to write an ‘import module’ with the sole purpose to import assets. Logically said this module allocates everything so they should have the ownership right? I’m inclined to say no. The information is retained by the engine and thus the proper way to design it is to request to the engine to allocate/free the memory. In practice this is implemented by a set of interfaces that are exchanged at module initialization.
At large many of the plagues that hunt dlls are just an natural extension to the problems of the language implementation itself and the eco-system around it. In particular you have to be really careful about code that compiles in the compilation unit of the callsite. This affects most template based classes. Odds are all of them are automatic memory management related i.e. container classes, shared pointers. This is because once they cross a dll boundary it’s just a matter of time until it horribly goes wrong by different heaps allocating/freeing the memory. For stl-related classes this is mostly circumvented by enforcing usage of the dynamic crt (/Md /Mdd) but still is something you should consciously consider when designing your own classes / inlined functions.
It’s plausible to properly tag everything with export/import attributes (implicit linking only). However it’s a solution that is highly demanding and most likely insufficient for larger code-bases. A more accepted solution is to avoid exposing state. By internalizing the state within dll boundaries either through interfaces or the pimpl-idiom this relaxes the exposed code quite a bit. In general interfaces causes twice the effort to maintain and author. Using proper coding-conventions it mandates two classes and two files which is why I personally favor pimpl as shown below.
Implicit and explicit linking usually doesn’t make much difference if the majority of the code that crosses dll boundaries uses virtual functions. In terms of exported symbols implicit linking allows you to simply tag classes with import/export attributes and will generate a library to that imports the symbols but also mandates the presence of the required dll with the expected filename at application startup. With explicit linking the application takes responsibility for importing the symbols and compatible class declarations. The main advantage of explicit linking is that you can choose yourself when to load or unload a module so generally used in plug-in architectures.
However to come back on the op’s original question(s). In visual c++ or rather said Windows itself heaps are created and destroyed by respectively CreateHeap, DestroyHeap. In the prologue or epilogue of the respective entry or exit points of an application/dll such heap is created or destroyed. To my recollection the handle of the heap drives malloc/free and likewise new/delete calls. Visual Studio installations accommodates the source code of the crt (common runtime) and much of this behavior could be altered if really desired.
That being said. Sharing heaps should be technically possible. The wisest thing is to ignore it and attempt to solve the complications of data ownership with reason, architecture and flair. For instance you might be tempted to write an ‘import module’ with the sole purpose to import assets. Logically said this module allocates everything so they should have the ownership right? I’m inclined to say no. The information is retained by the engine and thus the proper way to design it is to request to the engine to allocate/free the memory. In practice this is implemented by a set of interfaces that are exchanged at module initialization.
IMemoryManager* ms;
OnInit(IMemoryManager* manager) { ms = manager; }
malloc(…) { ms->malloc(…); }
free(…) { ms->free(…); }
At large many of the plagues that hunt dlls are just an natural extension to the problems of the language implementation itself and the eco-system around it. In particular you have to be really careful about code that compiles in the compilation unit of the callsite. This affects most template based classes. Odds are all of them are automatic memory management related i.e. container classes, shared pointers. This is because once they cross a dll boundary it’s just a matter of time until it horribly goes wrong by different heaps allocating/freeing the memory. For stl-related classes this is mostly circumvented by enforcing usage of the dynamic crt (/Md /Mdd) but still is something you should consciously consider when designing your own classes / inlined functions.
It’s plausible to properly tag everything with export/import attributes (implicit linking only). However it’s a solution that is highly demanding and most likely insufficient for larger code-bases. A more accepted solution is to avoid exposing state. By internalizing the state within dll boundaries either through interfaces or the pimpl-idiom this relaxes the exposed code quite a bit. In general interfaces causes twice the effort to maintain and author. Using proper coding-conventions it mandates two classes and two files which is why I personally favor pimpl as shown below.
__declspec(export) Class Thread { class Impl; char m_Data[32]; void Start(); }
Thread::Start( ) { reinterpret_cast<Impl*>(m_Data)->handle = CreateThread(…); }Implicit and explicit linking usually doesn’t make much difference if the majority of the code that crosses dll boundaries uses virtual functions. In terms of exported symbols implicit linking allows you to simply tag classes with import/export attributes and will generate a library to that imports the symbols but also mandates the presence of the required dll with the expected filename at application startup. With explicit linking the application takes responsibility for importing the symbols and compatible class declarations. The main advantage of explicit linking is that you can choose yourself when to load or unload a module so generally used in plug-in architectures.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
