  Shadow mapping & misc. work | 
Posted - 1/30/2006 11:12:42 AM | In the past days, i've implemented a nice little feature: automatic reloading of texture maps. The engine uses a texture manager, which checks every N seconds (currently, N=1) if a file image has changed ( i compare the size and the last modification date ). If this is the case, the texture is automatically reloaded on-the-fly. It's funny to load the viewer, open photoshop, modify a texture, and when alt-tabing back to the running viewer, see the changes take place without having to reload the viewer.
This week end, i started to work on dynamic shadow maps for the station. It's been really exhausting. I found two nice bugs:
- shadow mapping involves a texture projection from light space to world space. When applying the transformation to a vertex in world coordinates, i forgot to divide the projection result by the w component. The x,y and z components were invalid, and i didn't understand why, until.. too late.
- the second bug was a typo in the code. I applied a transpose() operation to a matrix, thinking it would act as selfTranspose(). In the first case, the matrix is untouched and the function results the transposed matrix, while in the second case, the matrix is erased by its transposed part.
Now, shadow mapping is more or less working. The results aren't very good yet, i'm afraid.. lots of artifacts everywhere. I still have to tweak them and to match the light frustum to the object frustum ( at the moment some resolution is lost in the shadow maps ). Each object ( the station is composed of 7 objects ) gets its own 1024^2 shadow map. It's funny to move the light in circles around the station, and see the shadows running around.
| |
| Tuesday, January 24, 2006 |
| Anisotropic lighting | |
Posted - 1/24/2006 9:37:33 AM | A coworker (Inigo Quilez) suggested that i use anisotropic lighting to simulate metallic surfaces on the space station. To my surprise, it works pretty well! It is also much more efficient than Cook-Torrance lighting, as it only requires to compute N.L and N.H, and to perform a texture lookup with these two coordinates. The lookup texture includes a small "diffraction" effect, which make rainbow colors appear. It is too strong now ( it should be subtle ), but it really gives a nice touch to the station. When seen in real-time, the anisotropic lighting is very nice, making distortions similar to caustics ( but more coherent and only viewpoint dependant )..
I've also implemented per-texture ambient occlusion ( as opposed to per vertex, which was limited by the tesselation quality ), and used 3ds max's render to texture feature to generate the textures. The ASE importer tool now supports multiple sets of texture coordinates, as i needed two ( one for the diffuse coords, one for the ambient occlusion coords ). I guess this could be used for lightmapping, too. Unfortunately, with all the textures and effects going on, ambient occlusion ( even per texture ) is almost invisible.
Mandatory screens of the station with anisotropic lighting:
If somebody still says that the station looks like plastic, i'll make myself a monk and go loose myself in the desert.
| |
| Station shading and lighting models | |
Posted - 1/22/2006 12:05:11 PM | I've been working on improving the shaders and lighting model used for metallic surfaces ( like most of the station ). In the mean time, Shawn has textured and added details to the top part of the station, with a biodome. It's funny because he actually started to work on it before Shadowstar mentionned the idea..
Meanwhile i've been implementing the Cook-Torrance lighting model, thanks to the articles and help of Jack ( Jollyjeffers on gd.net ). There was a few typos in the equations, but fortunately it didn't took too long to sort out the problems. Unfortunately, i found that tweaking the standard Phong-blinn model gave results looking as good as Cook-Torrance, at a much lower performance cost. The main difference with Phong equations is the addition of the fresnel term, but that's pretty much all..
The first screenshot is a view of the gas giant system from a high distance. Nothing special, but i don't think i posted one like that previously, so here it is.
The two other screens are from the top part of the station with the new shaders ( you don't wanna see the old one: it was really horrible ). I'm still not 100% happy with it, but it's definately an improvement. Some trees should have been visible in the biodome ( top-most part ) but i'm not handling transparent textures yet, so they're hidden.
| |
| Server emulation | |
Posted - 1/20/2006 9:50:48 AM | 
In the past days, i've been performing a network stress test, by emulating a server and a lot of clients connecting to it. In the above screenshot, you can see the test window as well as many statistics, which i'll explain in a short while.
The whole server is centralized around two components. The first one is, of course, the network module. I decided to code my own a year ago instead of using an established one ( like Raknet: i heard it didn't support a massive amount of connections, due to its per-packet overhead ). The result is INetwork, an implementation of the RDP protocol ( reliable-UDP ) with a few custom optimizations. It has a low cpu processing overhead, it automatically determines the quality of each connection in the system ( the "ping" ), can concatenate packets together to save header space, or even integrate acknowledgment packets into data packets. It supports many types of reliability, from fully unreliable ( pure UDP packets ) to pure reliable packets ( much like TCP ), with reliable but out-of-order packets in-between. I/O completion ports are experimental under Windows, but i found that the library performance was still excellent without it.
An interesting thing affecting the server performance is the quality of the network card. I already mentionned it in a previous entry, like 6 months ago, but some systems simply do not seem to be capable of handling a large amount of clients; the CPU usage rises extremely quickly, to struggle at around 100 connections. Fortunately, my test computer seems to have a good network card, and i went up to 2500 connections without too much troubles.
The second "core" component in the server is called IClustering. This is a small library which is responsible of determining which entities "see" which others. An entity is determined by a position in space, and a radius. This radius determines the minimum distance at which the entity is being visible by other entities. For example, the entity could be a tree and have a visibility radius of 1 km, which means that any other entity ( like a human ) that is closer than 1 km can "see" the tree. Note that the contrary isn't true: if the human has a visibility radius of 500 meters only, the tree will not "see" him. Each entity can be in 3 states: fully static, semi-static and dynamic. Only semi-static and dynamic entities can be moved inside the space ( or change their visibility radius ). Only dynamic bodies are informed about what they see or not. Technically, the implementation is based off a recursive regular grid, made of NxNxN cells ( each cell being a sub-grid ).
So, with these two components, i "emulated" a MMO server. To do that, i used many concepts:
- zones ( approximately 250 ): entities in different zones cannot see or interact with each other
- places of interest ( the green circles in the screenshot ): these are areas that mostly attract clients. In a Fantasy MMO, these would be cities, buildings or dungeons. In Infinity, they represent planets or stations.
- the clients: the yellow or cyan dots in the screen. They can be in active or idle state.
How it works:
- the server initially creates a number of zones ( 250 in my test ), with a set of places of interest in each one. The places can be recursively formed around other places ( to simulate moons orbiting planets in Infinity, or buildings in cities in other games ).
- a client emulator connects a defined number of clients per second ( between 10 and 100 in my test.. yeah that's a lot of connections/second, but the goal is to stress it after all ).
- each connecting client is affected to a zone. Zones have different probabilities of "interest" ( to simulate zones more busy than other ones ). The most active zones can gather ~30% of all the clients.
- the client is set into active state, and chooses a destination point. There are many probabilities for that. Around 30% to choose an existing place of interest. Around 50% to warp into another zone. Around 20% to choose a location where an existing client is. These numbers are for Infinity, where warping into different systems happens often.
- when a client arrives at destination, he is set into the "idle" mode. If he is warping into another zone, this zone is chosen based on zone probabilities, and is moved to a random point in that zone.
- when a client is in idle state, he has a 0.5% probability to be awaken and to choose a new destination point.
Network-related:
- there are normally 2 "update" packets per second ( to simulate sending position + orientation ), which are around 200 bytes each ( this is randomized a bit too ). These are sent in unreliable mode. They are sent to all the clients an entity ( another client ) is seen by.
- if the client is in idle mode, there are no updates.
- if the client is not seen by anybody, the update rate is lowered by a factor of 4 ( so one update every 2 seconds ).
- two updates per second can look low, but that's what Guildwars is using. And do not forget for Infinity, that ships cannot change of velocity quickly. This should be more than enough even during fights, thanks for dead-reckoning.
- when a client starts or stops to see another entity, a "creation" or destruction packet is sent in reliable mode ( ~300 bytes ).
- the clustering visibility computations are updated 10 times per second
Results:
- they are currently a bit biased, because i have a dual core machine, and the client emulator takes away a lot of resources from the server.
- some zones get a high amount of clients ( up to 150/zone ), most zones get a low amount of clients ( 0-5 ), the average being around 20 clients.
- the visibility radius of the clients is very small compared to the zone and places size. In consequence, most of time there are only packets sent when some clients are in the same place.
- for 1000 clients, the upload bandwidth is around 100 KB, and the download bandwidth around 200 KB. In average, 40% of the clients are idle. The average amount of bandwidth sent by a client to the server is 400 bytes, and the server returns ~500 to 1500 bytes. When many clients (10-15) are in the same place, this can rise to 5000-8000 bytes.
This post is becoming long, i'll add more informations/results later.
| |
| Tuesday, January 17, 2006 |
| Anti-hacking idea | |
Posted - 1/17/2006 5:24:09 AM | I've been thinking to a solution to prevent hackers from releasing a pirated version of the game giving unfair advantages. Surely all the critical data and calculations will be done on the server side, but protecting the client from "cheap hacks" (like aimbots) is necessary too. This is for an online game, client/server based.
So here is the idea: each time a client wants to connect to the server, it has to download & replace the client EXE from it, pretty much like a patch.. but every time it connects. The trick is, the downloaded EXE is different each time and incompatible with its previous version, including the network protocol, function adresses, etc..
When the server detects an incoming connection, it picks up one EXE from a pool (which can be arbitrarily as large as you want) and sends it to the client. All the EXEs use a different network protocol and once the download is complete, the server is set to only understand the specific protocol corresponding to the client. If there's still no reconnection after 10 seconds, for example, the process has to restart from the beginning.
Some drawbacks i can see:
- additional bandwidth (if the EXE is 1 MB, that's 1 MB of bandwidth consumed each time a client connects).
- the server has to understand all the possible protocols at the same time
- the number of client EXEs has to be pretty large
- each time the client EXE is patched/improved (by the developpers), many versions have to be recompiled.
- not 56K friendly. ADSL is not problem because downloading 1 MB is only a couple seconds.. but 56K modem users will have to be patient.
The advantages:
- if a hacker modifies his local EXE, the server will send him a new one when he connects. If the hacker ignores the newly downloaded EXE and tries to connect to the server with his hacked version, the protocols will be incompatible and the connection refused.
- if the hacker downloads a new EXE, wants to hack it and connect to the server with this hacked version, he has to do it in less than 10 seconds.
- a hacked version cannot be redistributed because it would be invalidated as soon as average-Joe connects to the server.
The flaws..?
- could the hacker modify the EXE in less than 10 seconds ?
- still does not fix the content-modification problem (like modifying textures/models or external DLLs).
Thoughts..?
| |
| Station screenshots 2 | |
Posted - 1/15/2006 1:38:55 PM | Not much to say - a few fixes and details by Shawn, but code-wise, most of my work this week end has been oriented towards the Minas Tirith patch.
In the last shot, you can see the station ( which is in orbit at an altitude of 300 Km ) looks like from the ground.. definately huge. Atmospheric shading missing.
| |
| Space station screenshots | |
Posted - 1/13/2006 3:23:40 PM | As promised, here are the first screenshots of Shawn's space station:
| |
| Thursday, January 12, 2006 |
| Station shaders | |
Posted - 1/12/2006 2:17:17 PM | I've been pretty busy this evening to get the shaders ready for the space station. Shawn is making nice progress on it, but it's far from being finished. The results are starting to look nice, so i think i'll be posting a few screenshots of how it looks inside the game tomorrow.
I've implemented a couple effects in the shaders:
- per-pixel lighting with bump mapping
- per-pixel specular lighting with gloss mapping ( making some areas more or less shiny )
- self-illumination ( like small windows ); but i haven't done the glow filter yet, so the contours look sharp.
- ambient occlusion ( still per vertex ), subtle because not very visible.
In the coming months, i'll also implement:
- dynamic shadow maps
- dynamic cube map reflections
- ambient color reflection ( via a small blurred cube map )
At the moment i'm trying to get all the shaders working in object/tangent space, to be able to position and rotate them arbitrarily in world space. The amount of code to modify is very small, but it's very tricky.
| |
| Tuesday, January 10, 2006 |
| Ambient occlusion | |
Posted - 1/10/2006 6:02:10 AM | I've been working on an ambient occlusion module for my ASE2Bin model viewer, and tested it on a Minas Tirith building.
There are many issues with the way it is currently done. The main one is, while waiting for a proper lightmapping implementation, the ambient occlusion values are stored per-vertex in a color component. And per-vertex means being very tesselation dependant. Unfortunately, the building below is on the "good" side; i've seen many buildings that didn't behave as well.
Initially, the algorithm casted for each vertex 256 random rays in the half hemisphere pointed by the vertex normal. Each ray intersects the scene, and the distance is used to compute an "ambient" color. All the results of the rays are averaged, and the end result is a color to store per-vertex. That didn't work very well, because often, a vertex is "inside" a wall, while it was logically on the wall, but the modelers prefered to do it that way to save polys and avoid T-Junctions.
I've experimented an alternative algorithm, based on casting rays for each triangle, instead. 256 rays are still thrown into the scene, but this time from a circle more or less on the center of the polygon. The ambient color of a triangle is then added to its 3 vertices, and in a next pass, all the vertices ambient values are averaged. The result is looking much better, but it's still tesselation dependant, and i don't think there's any way to fully fix it.
Here's a result on a single building:
LEFT: diffuse lighting only
RIGHT: diffuse + ambient occlusion
| |
| Minas Tirith release 10 | |
Posted - 1/8/2006 11:35:25 AM | The latest version has been released.
Head to the Official Website and download the realtime viewer. System requirements are high, so i would not advise that you download it unless you have at least a Gigabyte of RAM, a 2 Ghz CPU and a 128 MB video card.
For those of you who do not meet the system requirements, i've made a video showing off physics, collisions, first-person walking mode, etc.. It's 6:30 and encoded as usual in Divx4 (50 MB):
Video
And here is another video, low quality version (25 MB), showing off things a bit differently (the beginning is the same though): Smaller video
| |
| Saturday, January 7, 2006 |
| Minas Tirith release | |
Posted - 1/7/2006 7:37:29 PM | The new Minas Tirith version (10) should be released tomorrow. I've been spending an incredible amount of time today to be ready, and i feel exhausted. This new release should include first-person camera walking/running with full collisions, jumping, and most of all.. physics! You can now make statues fall down, open some doors/gates (just by pushing them), hit candleholders.. I've implemented a cannonball (throwing little spheres), too. It's funny to see balls bounce on the staircases, follow slopes, hit walls, etc.. On the technical side, i've implemented everything with ODE, and it went pretty smoothly, except for a few weird bugs. The worst bug was basically generating invalid normals (NaN) at some contact points, and i found out that it was cause by degenerated triangles (null area) in the dataset. Filtering out these triangles at loading time fixed most of these issues.
This new version will require a lot of memory: no less than 512, one gibabyte is recommended, and 2 GB would be ideal. The city is now close to 5 million polygons, and with 10 levels for LOD, the index arrays data is starting to weight a grand 350 MB of ram. Collision is using a nice amount of memory, too. I'll certainly release a video tomorrow to show off the physics, for those who do not have enough memory or a good graphics card.
| |
| Back to programming | |
Posted - 1/3/2006 10:04:28 AM | Happy new year 2006!
I've back to do more programming. I've been trying to implement collision detection in ODE for the Minas Tirith project (which incidently, is using Infinity's engine: combo hit). It didn't work quite well. As soon as two objects were colliding, they ended up with a NaN (not-a-number) position/velocity. I tracked down the problem into ODE to find a very strange thing..
Picture this: a structure dContact, contains collision detection informations about the contact, its position, normal, penetration, etc.. I've got an array of these (in C++: dContact contacts[64]). Easy, huh ?
Hell no: typing in the debugger sizeof(dContact) returned 196, while sizeof(contacts[0])) returned 200. An offset of 4 bytes, coming from nowhere, while the two are "logically" equivalent.. how is that possible ?
Well, the thing is, it's perfectly possible. After spending a couple of hours debugging that thing, i found the cause of the problem: in another DLL, a header file was setting the pack alignment of structures to 1, then setting it back to the default, 4. Excepting that.. the default in VC is not 4, but 8 :) A stupid mistake which caused the dContact structure to be defined virtually twice: one with a pack alignment of 4 (the one included by the application, which is using the other DLL); and the one with a pack alignment of 8, included by ODE.
Cool, isn't ? I love programming.
In other news, i've received a first version of the space station Shawn is modelling. It's full of mapping errors, but as it appears correctly in 3ds max, it means i've got a serious bug in my exporter code. That's weird, too, because i've exported tons of models with that exporter before, and the texture coordinates has always looked good. More debugging to come.. sic.
| |
|
| S | M | T | W | T | F | S | 1 | 2 | | 4 | 5 | 6 | | | 9 | | 11 | | | 14 | | 16 | | 18 | 19 | | 21 | | 23 | | 25 | 26 | 27 | 28 | 29 | | 31 | | | | |
OPTIONS
Track this Journal
ARCHIVES
October, 2009
August, 2009
July, 2009
May, 2009
April, 2009
March, 2009
February, 2009
January, 2009
November, 2008
October, 2008
July, 2008
June, 2008
May, 2008
April, 2008
March, 2008
January, 2008
December, 2007
November, 2007
October, 2007
September, 2007
August, 2007
July, 2007
June, 2007
May, 2007
April, 2007
March, 2007
February, 2007
January, 2007
December, 2006
November, 2006
October, 2006
September, 2006
August, 2006
July, 2006
June, 2006
May, 2006
April, 2006
March, 2006
February, 2006
January, 2006
December, 2005
November, 2005
October, 2005
September, 2005
August, 2005
July, 2005
June, 2005
May, 2005
April, 2005
March, 2005
February, 2005
January, 2005
December, 2004
October, 2004
September, 2004
August, 2004
|
