Unreal Engine... How did they do it?

Started by
20 comments, last by dougbinks 11 years, 6 months ago

<snip>


You hit the nail right on the head, this is exactly the thing I wanted to experiment with as I too am using specialized allocation schemes in my engine code which may not work too well with this system, and I really don't want to give them up to implement this feature.
I can see the destroying and reallocating resources in memory becoming a problem since I mostly use a contiguous allocation scheme so I can easily group resources of one type into one contiguous chunk of memory (no fragmentation + data-oriented design = happy cache + happy programmer). Cleanup is done in chunks too as it pretty much comes down to rolling back to a base point in memory and calling the appropriate destructors in order, so selective destruction of data is not really recommended - I'm not sure whether this will actually pose a problem, but it's definitely something I need to keep in mind

I would be interested to see this run on consoles, I can imagine it being a major productivity boost if developers can make on-the-fly changes to code running on a console.

I gets all your texture budgets!

Advertisement
I would be interested to see this run on consoles, I can imagine it being a major productivity boost if developers can make on-the-fly changes to code running on a console.
Lots of companies do this already, but only for procedures, not class-layouts. The PS3 encourages a "job based" programming model, where your procedures are more separate from your data-layouts than in traditional OOP, so it's not too hard to shoe-horn in code-reloading to systems written this way.
Reloading a DLL (equivalent) and changing some function-pointers is simple. It's only the reloading of class-layouts (and the implied serialization/deserialization/pointer-patching) that complicates matters.

As far as I'm concerned, every engine should already be doing this for their (HLSL or equivalent) shader code and any "script" code (e.g. Lua) too ;)

[quote name='Radikalizm' timestamp='1344410766' post='4967284']I would be interested to see this run on consoles, I can imagine it being a major productivity boost if developers can make on-the-fly changes to code running on a console.
Lots of companies do this already, but only for procedures, not class-layouts. The PS3 encourages a "job based" programming model, where your procedures are more separate from your data-layouts than in traditional OOP, so it's not too hard to shoe-horn in code-reloading to systems written this way.
Reloading a DLL (equivalent) and changing some function-pointers is simple. It's only the reloading of class-layouts (and the implied serialization/deserialization/pointer-patching) that complicates matters.

As far as I'm concerned, every engine should already be doing this for their (HLSL or equivalent) shader code and any "script" code (e.g. Lua) too ;)
[/quote]

I see, interesting. I don't have any experience whatsoever with console development, unless you count XNA + X360 development as console experience, so this is new to me :) Sadly console development is one of those things you can't experiment with on your own...

As for shader and script code my system does in fact allow for runtime editing and recompilation in development builds, just like it allows for on-the-fly editing and reloading of any other type of resource, and you're absolutely right about it being a crucial technique if you ever want to get anything major done.

I gets all your texture budgets!


[quote name='Hodgman' timestamp='1344412668' post='4967294']
[quote name='Radikalizm' timestamp='1344410766' post='4967284']I would be interested to see this run on consoles, I can imagine it being a major productivity boost if developers can make on-the-fly changes to code running on a console.
Lots of companies do this already, but only for procedures, not class-layouts. The PS3 encourages a "job based" programming model, where your procedures are more separate from your data-layouts than in traditional OOP, so it's not too hard to shoe-horn in code-reloading to systems written this way.
Reloading a DLL (equivalent) and changing some function-pointers is simple. It's only the reloading of class-layouts (and the implied serialization/deserialization/pointer-patching) that complicates matters.

As far as I'm concerned, every engine should already be doing this for their (HLSL or equivalent) shader code and any "script" code (e.g. Lua) too ;)
[/quote]

I see, interesting. I don't have any experience whatsoever with console development, unless you count XNA + X360 development as console experience, so this is new to me smile.png Sadly console development is one of those things you can't experiment with on your own...

As for shader and script code my system does in fact allow for runtime editing and recompilation in development builds, just like it allows for on-the-fly editing and reloading of any other type of resource, and you're absolutely right about it being a crucial technique if you ever want to get anything major done.
[/quote]
I have not really used XNA for 360 but I think there will be a few features the runtime might not expose to keep the writing of game code simple. I am thinking of manually setting which tiles to clear on the backbuffer and setting up tiled based rendering in particular, but I don't know this for sure to be honest.

Worked on titles: CMR:DiRT2, DiRT 3, DiRT: Showdown, GRID 2, theHunter, theHunter: Primal, Mad Max, Watch Dogs: Legion

I think this feature has nothing to do with visual studio. VS is only here bound to their system to allow for nice editor features. but compilation/link/relaunch is handled by them.
Surely, it cannot work for everything, you cannot modify core code, it must be within precise "plugin" architecture, and surely, like android, there must be a lot of constraints. You mentioned before something about serialization, this is exactly what android requires of all application because they can be killed at any time.
it is easier to rebuild a game logic that says "when space hit then jump" which requires no data, than rebuild the code of the deferred pipeline because it would require rebinding all uniforms, rebuild all render targets etc etc.
But of course, it is still amazing.

I think this feature has nothing to do with visual studio. VS is only here bound to their system to allow for nice editor features. but compilation/link/relaunch is handled by them.
Surely, it cannot work for everything, you cannot modify core code, it must be within precise "plugin" architecture, and surely, like android, there must be a lot of constraints. You mentioned before something about serialization, this is exactly what android requires of all application because they can be killed at any time.
it is easier to rebuild a game logic that says "when space hit then jump" which requires no data, than rebuild the code of the deferred pipeline because it would require rebinding all uniforms, rebuild all render targets etc etc.
But of course, it is still amazing.


You might want to read the slides of their presentation (it's posted at the link I provided), it'll clear some things up about the inner workings ;)

I gets all your texture budgets!

Some great commentary and questions here, here's some responses which I hope clarify things:

@Lightness1024 and Radikalizm:

We're using Visual Studio as our compiler, since that's the most common compiler used for games, and there's a freely available Express version which we also support. Other compilers and OSes could be supported but would require a little extra work. We don't use any specific features of VS for the base implementation, though the pragma comment library feature is an easy way to get libraries added - but we intend to supply a cross platform version eventually. In particular I'll likely port to Mac OS X and Linux at some point, though that may not be for a few months due to other work.

@Hodgman

Myself and the other authors are all professional game developers with console experience, and a similar implementation at Crytek is working on the XBox 360 (see our develop conference slides linked from the blog). Memory is certainly an issue for console development in general and this technique would add some overhead, though there are development consoles with more memory to aid with this.

Many changes could be done with in-place serialization or even serialize to disk/network etc. and back so as to be able to free up memory prior to recreating the objects and thus also handle fragmentation issues. Additionally, though in principal all data types could be runtime changed (for example I've used the technique for graphics coding), most of the larger asset types are likely to have a very stable memory format and so not need runtime serialization.

We may make a client/server style runtime compiler for systems like iOS, Android and consoles at some point, but we'd not be able to open source the console work due to licensing conditions.

Hope this helps, I'll try and monitor this thread and try to answer questions that come up.
:) thanks for replies.
but all that seems to me a little scary. trying to maintain heap while rebuilding code in a non-managed/non-interpreted language :s
what happens if a plugin creates a bunch of instances of structure S and we change the content of structure S to a new layout of data we will name S2, then after rebuild, if the runtime tries to access those old instances the code will think to see S2 structures but actually those are S structures in memory. This will corrupt the content and crash if it involves pointers.
how would that be handled ?
It's actually a lot easier than you'd think to use, and we have an error handling mechanism to enable you to correct these problems and keep working. Even if the game crashes in a completely fatal way, you're no worse off than if you'd made an error in your code, compiled and ran it as normal. If data gets corrupted, you could re-load a save point (currently our demo doesn't create save points when loading new code, but it would be trivial to add).

Best way to find out if it works for you is to give the demo a try.

Note that our primary target is programmers - who are used to dealing with bugs and crashes and the state they create is code which doesn't get lost when the program crashes. If you want to use this for designers (and we believe it can be, see latest post on the console for one example), further error protection methods such as the above mentioned save points should be used.
If anyone is following this thread still and using OS X, they might like to know we've recently added OS X support (only tested 10.8 and XCode 4 Clang++ so far). Would appreciate any feedback.

This topic is closed to new replies.

Advertisement