a) you need a memory management system that is cross-platform compatible as you would possibly have a heterogenous network of machines working together.

I suggest you 'tag' memory objects and type them, that you restrain yourself to working with 32-bit quantities (aka 'long' and 'float' types) because of byte ordering and packing efficiency.

Accessing a memory object is a "Lock / Modify / Unlock / Network Flush" combined operations. Remove the "Network Flush" part for just reading. "Lock" should pull the object from whatever computer it is on if it's not in local memory. Written in C++, you can have accesor functions for this.

You should design your tag numbering system so that it puts the owning computer's number into it. For example, reserve the first 8 bits of the 32-bit tags for the network computer network. Then you can split the remaining bits into buckets. For example, tag 0x1234ABCD could translate to location "(void *) &memory[0x12][0x34][0xABCD].pData". You can make your data types match the second bucket to that for example all cameras are allocated within the same memory area; it's more heap-efficient. In the example above, cameras of computer 0x12 on the net would be located in the 0x34 memory area, and the exact camera you are looking for is camera # 0xABCD. Makes type checking and memory management much easier on the system and on the network.

Scene data should be organized as a DAG whereby an element can reference other sub-elements by tag. You could have special "group" data tag elements that contain a variable list of tag children and an "object" tag element. This way you can have hierarchical data trees.



b) you need a task queue manager. A "task" would be a description of a rectangle area of the resulting image, a thread would be responsible of computing.

You can use the tag-based system described above to describe task elements and relationships to other tag-based tasks. When the system starts, all system and network threads are waiting for the task queue manager to release tasks. The main thread fills up all the rendering tasks into the tag-based memory and works out the relationships between the tasks. Then it send the signal to start processing at which point all threads in the system start pulling tasks one at a time.

Tasks can be split into sub-tasks and put back in the queue; for example, you can split a large Bezier primitive into ever smaller Bezier patches until a set criterion is reached whereby the Bezier patch is replaced by two triangles forming a rectangle. The task manager should be smart enough not to release a task for processing if its dependent task have not been resolved. The task queue thus has to do a topological sort each and every time a new task is either removed or added, and this operation shall be atomic (aka thread-locked).



c) you need a way to exchange data with other computer in the net that share that entire distributed database and a way to represent a remote computer. You can add a "network thread" for each computer you are connected to, and this thread's job is to exchange data with its assigned computer back and forth and act as a worker thred on the local system for local resources. For example, if a remote computer "Locks" a tagged data block this should translates as a TCP communication mechanism with its local thread representative and this local thread will thus have to secure that memory local on the local computer by performing a local "Lock / read / Unlock" operation on behalf of the remote computer.




d) you need a way to reference outside resources that would be available across the net. For example, it doesn't make sense to transmit picture data back and forth if there is a central local for all images used for rendering.

I suggest you use a sed-like naming scripting system whereby each computer on the net preloads some pre-defined tokens specific to the platform and relative location on the net. Some of the tokens could be based on a small file that is available locally or on environment variable. For example, let's say the token "Project" is associated with the string "Raytracer/version 3" and that each computer has defined its "IMG_LOCATION" environment variable to point to the network path where the image library exists. Then the following filename:

${IMG_LOCATION}/projects/{Project}/texture.jpg

would translate to

/server/nfs/projects/Raytracer/version 3/texture.jpg

on one system running Linux, and

//lnxsvr/samba/projects/Raytracer/version 3/texture.jpg or
I:/projects/Raytracer/version 3/texture.jpg

on, say, a Windows XP system.

Note that if you implement shaders that are packaged as DLLs, then the token name "DLL" could be mapped to "dll" on a Win32 machine and "so" on a Linux one.



Hope this helps.

e) you could investigate using a scanline rasterizer for primary rays. For each pixel on the screen you can quickly determine the list of surfaces intercepted and shade them in reverse depth order. An obvious optimization would be search for the first opaque intersected object fragment in the list and start your shading from there instead of the very end of the fragment list. Shading the fragments become "tasks" whereby the top task is the final color computation and all the other fragments undernearth are dependent tasks to be resolved first. This helps your "lazy evaluation" paradigm you were looking into.

f) you can assign a priority in the rendering tasks so that the bucket in the center of the screen are rendered first. It's easier to spot a mistake if what it is you are raytracing is in the middle of the screen and the raytracer DOESN'T start from the upper left corner.

g) you should implement a "kill" function in the task queue manager so that the queue can stop distributing tasks and ask running threads to abandon their current work and revert to the "wait" state.

h) an interesting optimization for the task queue manager would be to assign redundent tasks to network threads that would be otherwise in a wait state. This happens then the last tasks are distributed as the rendering processing is about to end; last thing you want is for the slowest computer to hold up the entire process. This optimization allows the best available computer to "win" and kill all the other identical tasks being run.

Quote:Original post by LostAgain
...I am trying to write a simulator for a large kinematics engine i am designing the kinematics system is mostly finished. now for the rayracer...

If the main project is your “kinematics engine” you can use a free high quality rendering system like NVIDIA Gelato 2.0.

An example(c++):

The result:

Rendering time : 1.46s

Depending on how set you are on writing your owd distributed computing system, I may be wasting your time, but I thought I should speak my mind. You could always use Pov-Ray combined with an OpenMosix Linux cluster to do your rendering. You could create a cluster very simply with something like ClusterKnoppix (if you don't have PXE enabled NICs, there are simple ways areound it). I believe Pov-Ray has many of the advanced features that you would ike and could be very powerful when combined with OpenMosix. In order to tie your kinematics simulation in with Pov-Rays rendering engine, you could write a program that rewrites the code based on which frame needs to be rendered, then renders with povray...The clustering should be made quite simple with OpenMosix.

Quote:the reason i am doing depth first tracing instead of breadth first is because i would like the clients to ONLY down load the information they need to check rays in there small area

It doesn't make any difference if you do depth-first of breadth-first unless you assume a very low degree of scattering. What you describe as your typical scene isn't going to bode well with algorithms that assume ray coherency (ex: Arvo & Kirk's Ray Classification scheme). In a system with any sort of scattering, any pre-processing step needs to have access to the whole scene data; at best you could compute per-object bounding boxes and tesselate on-demand, and redeem the tesselation memory on an LRU basis.

Quote:I am trying to create a rather robust work client server set up as it may have to work for another project

You can also look up MPI (Message-Passing Interface) as a more standardized alternative than the tag-based one I described above.

Quote: motion blur

Just curious as to how you plan to deal with this one. Adaptive tessellation doesn't bode well with motion blur in general as you cannot guarantee frame-to-frame topology coherency. The way commercial system do this is by computing per-vertex motion vectors and shoot stochastically time-sampled ray onto the moving triangles. But that generates artifacts if the frame rate is below an object's deformation Nyquist limit (i.e. the triangle motion paths cannot be approximated well-enough by a line within two frame samples). A way around this is to not only have access to all the geometrical data for the scene at that frame, but also of all the animations. Therefore the animation can take a non-linear path and be more precise, but that would be a lot more costly. As you don't really care about computation costs, that's probably the way to go.