I have not added anything to this topic yet, as what I would say has already been said, but more eloquently .

I just have one query regarding YogurtEmperor's post, I would like to say that I like how your system sounds, you seem do alot the same way as I, also you have done many things the way I intend to do them, you know when, I get around to it.

But regarding..

I use a multi-map because my keys are file names and checksums. ........stuff.............. Most textures will have file names associated with them, so I can usually prove a texture to be unique or not immediately.

Are you saying you check through the pixels of a texture to determine a match? Oh I'm assuming that the textures without file names are generated or composites. If that is the case then would you not just give these textures, a fake filename, based on their construction?
Maybe I'm misreading but this is the only place where I have disagreed with specifics on your implementation.
Oh and yes I realize that this is a rare case, but still seems like a bad practice to moi.

[font="arial, verdana, tahoma, sans-serif"][quote name='YogurtEmperor' timestamp='1315902111' post='4860992']Instances share the master models, and the master models share textures between each other.

Like a texture atlas of some sort?[/font][/quote]
Texture atlasing is a different area of optimization apart from the resource manager.
I haven’t had time to add this yet, but the plan is that atlases could be constructed on a per-master-model basis, so the textures within a single model could become atlased. This would always be done offline. Then once again that creates a nameless texture that once again will be referenced by checksum and shared. Another model with the same atlas embedded into it will use the existing resource instead of uploading a duplicate.

I have not added anything to this topic yet, as what I would say has already been said, but more eloquently .

I just have one query regarding YogurtEmperor's post, I would like to say that I like how your system sounds, you seem do alot the same way as I, also you have done many things the way I intend to do them, you know when, I get around to it.

But regarding..
[quote name='YogurtEmperor' timestamp='1315962831' post='4861335']
I use a multi-map because my keys are file names and checksums. ........stuff.............. Most textures will have file names associated with them, so I can usually prove a texture to be unique or not immediately.

Are you saying you check through the pixels of a texture to determine a match? Oh I'm assuming that the textures without file names are generated or composites. If that is the case then would you not just give these textures, a fake filename, based on their construction?
Maybe I'm misreading but this is the only place where I have disagreed with specifics on your implementation.
Oh and yes I realize that this is a rare case, but still seems like a bad practice to moi.
[/quote]
Checksums don’t actually verify a match, so I need some way to be sure. Memory compares are a bit slow, but in most scenarios it will never get to that.
However I have been considering an alternative that is still not 100%, but theoretically close enough: Double checksums.
I plan to make a second checksum routine. It is unlikely that any texture that passes both checksums is not a match. If I integrate this, I will keep the final memory compare as an #ifdef option, so that engine users can activate it if they ever encounter a problem with the double checksums.


L. Spiro

I don't understand this memory compare thing -- if you're able to perform a memory compare, then you've already got two copies of the texture in memory, right? How does that occur in the first place?

Yes, but only once per file (though I agree it is annoying anyway).

#1: Embedded media already have a checksum (soon to be a double checksum) so these do not need to be unpacked unless they are unique.
#2: But files on disk currently need to be loaded if their file names are not duplicates. They have to be checked against unnamed media, so they are loaded, checksums made, and checked against unnamed images. If a match is found, the unnamed media is then given the name of the file so that it cannot cause a useless load more than once, and the file from the disk is thrown away. I believe it was Socrates who said, “It is better to have loaded and lost than to have overloaded.”

When I finish my toolchains I will fix #2 by having each game be a project, and building the game is done entirely (or nearly) through the project tool. The user will add textures to the game through the project tool, and that will give me a chance to create a dictionary associating file names with double checksums. This way I can perform the double-checksum without loading the file.


Luckily it is rare that a file is a duplicate of embedded media, so the problems with #2 actually don’t occur too often. That is, when a file is loaded from disk, it is very rarely immediately tossed out.
In addition, post-load duplicate-discovery can only happen once per file, so although this method can be streamlined by a complete pipeline, it still works well in practice.


L. Spiro

I was meaning that for any resources that do not have real file names you just give them a fake one. Now for me textures without file names are either generated random noise,
normal maps generated from height maps or when several source images are packed into one. I make up file names for these anyway as I will cache the result, (later this will be improved to only caching appropriate files, but currently everything is cached).
I'm assuming your resources without file names are of similar make up, so I if you makeup file names for these you can have an exclusive test without a memory comparison.

On a side note I would be very interested to see what too different textures could actually satisfy to checksums.

That is what I did in my first game engine.
The problem is, how do you uniquely identify the new texture?
Here are the thoughts that lead me to my current implementation:

Let’s say you combine a diffuse texture and an alpha texture on a given model. The media was originally embedded, so there is no file name for either texture.
Even embedded media have names, so you could combine the names of the textures to create the name of the new one.
suit_diffuse + marble_alpha = suit_diffuse|marble_alpha

That makes a unique name within the scope of the textures referenced by that model.
What about another model? It could make the same name using different textures.


So to avoid having another model overwrite your texture, you could add the name of the model to the texture.
So ship0|suit_diffuse|marble_alpha will not be overwritten by ship1|suit_diffuse|marble_alpha.


But then what if ship0|suit_diffuse|marble_alpha and ship1|suit_diffuse|marble_alpha are actually the same? They should be shared, but instead they will be duplicated.

What about a texture that was embedded (no file name) and then what if that same texture is loaded from disk? Unless I randomly generated the disk file name for the unnamed embedded file, I would have a duplicate on my hands.


I realized I needed a naming system that is not based off how the texture came into existence.
What is the only way to really know if two textures are the same?
Memory compares!
*cough*
So what is the next best way?
Checksums!


If all of your files are coming from disk, you can rely on combining file names to create unique names. And noise textures are unlikely to be duplicated so you could just generate random characters for their names.
But if you are using embedded media or for any other reason will not always have a texture file name handy, you need to be careful about how you uniquely identify textures.


L. Spiro

I might..might quite possibly have overlooked textures of the embedded variety, thanks for clearing that up.
I'm going back my corner now.

P.s I was rushed on my earlier posting, did not have time to give due kudos for "I believe it was Socrates who said, “It is better to have loaded and lost than to have overloaded.” " was quite good.
P.s.s @CombatWombat - If your still here, sorry for derailing the thread, I kind of assumed you resigned from it.

I don’t think the topic has been derailed. Even if the original poster does not need such a complicated system, there is a lot of useful information here from many people.


CombatWombat, read this.
You should definitely submit information for a render queue to sort, but keep it minimal to avoid transfer overload. If you use the sorting method I proposed there, sorting will be the same speed regardless of how large the render queue data is, but still you do have the initial upload of render data, and in that area bigger = slower.

So you don’t want to send whole matrices to the render queue. Some people offload the entire render to the render queue. It helps them to avoid a virtual function call which is costly on PlayStation 3, but they have to send more data which is slower.
In my implementation, I felt that sending matrices would be slower than invoking the wrath of a virtual function, so the render queue does not receive a matrix and instead delegates the final render task to the object that submitted the render queue packet.
Since the submitter of the packet will be responsible for the final render anyway, I can reduce the amount of data I send to the render queue even further by not sending pointers to index buffers and vertex buffers.


So what do I submit? Well, logically I submit only what is important for the actual sorting process. That is the whole point of the render queue after all.
As mentioned in my article, texture uploads, shader swaps, and render states need to be kept to a minimum, so this is the type of information I send to the render queue. I send a shader ID, a set of texture ID’s, and some render states.

After the render queue sorts these packets, it runs over objects in sorted order and calls a virtual function on the objects that sent the packets to perform the final render. This means of course that I also submit a pointer to the object that sent the packet.


L. Spiro