• Advertisement
  • Popular Tags

  • Popular Now

  • Advertisement
  • Similar Content

    • By Elwin
      So I've started in learning OpenGL , then I found the NeHe website and its Legacy Tutorials  (here is the link  http://nehe.gamedev.net/tutorial/lessons_01__05/22004/ ).But the problem is that I can't download any   examples of code ( neither C code examples nor any other language ) .As you see it is impossible to download http://nehe.gamedev5.net/data/lessons/pelles_c/lesson01.zip . Does anyone has expamles of C code of this lessons ?  And the second question. If I prefer coding on C , should I choose GLUT but not an OpenGL ?
    • By Anthony Cao
      Hiiii, so ive been studying marketing in my own time and I have experiments i wanna try do to past recorded strategies of other marketing strategies. Also im on Redbull, Dr Pepper and a 48 hour energy mixture soooooo excuse my preppy nature on my typing and grammer usage. 
      As far as I can see I have a theory (actually I got hundreds of these), so as far as I know a game doesnt necessarily need to be advertised to be sucessful from the content publishers themself. There are other means to counter the forceful advertisment. Yea im not speaking of reviews and playthrews to go up in search ranking and relevancy tho that is a tactic that happens on its own. 
      Sooo whats on my mind today is that online games pretty much baits the players to spend money to look smancy, now what if the fancy panties were to be grabable in game without the need to spend actual money. Then no one would spend anything right? What if i told you there is a way to trick players mentallity using this method in order to get them to spend money. Your probably thinking to yourself "go to sleep Anthony you are drunk"... Im energy high theres a different... If a function were to be available in game for free then by all means players would go the free route, now if the free route was difficult to achieve the player would quit the game and you will have a fanbase on those rage quiters (Dark Souls, and binding of isaac). If the free route was easy to perform yet very time consuming and grindy then thats more hours spent on the game. Now heres the trick, what if the option to purchase instead of grinding was available? 
      Then the players would think, would they spend a full hour trying to fetch the item, which in a working net worth terms 10$... Or would they rather spend 3$ to get the item right away. 
      [deleted by moderator]
      This weird drink mixture is making me lo key depressed. 
      Alsooo what do you think about this marketing concept... 
    • By a.gene
      My name is Aaron. I have been a software engineer/ programmer going on 8 years working with languages such as C++,C#, Java, JavaScript. Most of my development experience has been backend development. I always wanted to get envolved in gaming development just never found the time. I am a quick learner and looking for a experience where I could work with a team to learn about game development and also help create cool games.
      I could contribute 15-20 hrs a week mostly in the evenings I am currently located in Washington DC. If you have any projects that you are looking for some help on or think I have skills that could help please don't hesitate to reach out.
    • By Rhia.Abstrakt
      With my design document relatively fleshed out, my mind is turning towards starting a prototype. As they say, the first step is always the hardest. This will be my first self-driven project, and that first step is looking awfully daunting. Does anyone have any advice on where to start? I know this is a very personal choice, but I'm interested to hear how more experienced devs approach that first step in development. For example, do you start small with something like the player movement and controls? Or do you prefer to set the stage a bit first by establishing a bit of the world space?
    • By Dave Haylett
      Hi all,
      I’ve nearly finished my C# WPF project (using VS2017) and am now thinking about deployment and managing updates. This is my first time doing this, and I wanted to run my current thinking past you to see if you can help, as I don’t think my current solution will fully work. I have drawn a quick mock-up of what’s in my head at the moment (attached).
      My app needs to be able to update itself, but also to add in new content once I create it (this is in the form of small .zip files, ~100kb each, and due to the volume of these files [currently 3000, hopefully expanded up to 5000 in a years’ time] I have kept these separate from my project, i.e. not as Content). I welcome comments on this decision.
      Due to keeping the content separate, this kind of broke my initial intention of “simply” using ClickOnce deployment, and letting Windows manage the updating for me, and so I’ve been thinking of how to manage i) initial app deployment, ii) ongoing updating of the app, iii) ongoing addition of new content.
      I’m currently intending to use my *free* DropBox account to host all the files, I think this is possible. Does anyone know of any better web hosting I could use?
      If you could refer to my attached diagram, what I’m currently thinking of is:
      Deploying the initial version of the app using ClickOnce by Publishing to DropBox or wherever (my project A publishing to web host D). Users can then download the Installer from here, and install on their local machines as usual. This initial installation wouldn’t have any content (the 3000 zips) and so these would be pulled down to the local machine by the below step (the initial batch of 3000 zips would only need to be downloaded the first time the app is run, after then it’d just be new content that’d be downloaded):
      Also, I have as a second Project in my overall Solution a small Updater app, which when run by the users will connect to DropBox and pull down an xml manifest of my project files and the content (I create this manifest myself), and synchronise any files which are newer. This Updater app will be downloaded and installed by the users too. In the diagram, I publish Updater app code C to web host G, downloadable by users. Please see below question:
      Question: my main app and my updater app are two separate projects under the same Visual Studio Solution. Is it possible for me to build/publish these together so that the two executables are together in the same output (in the same folder)? The reason I have them separate is so that the main project’s executable file and any resource files can be overwritten by any new files pulled down by the Updater app without the Main app being open and locking these files out.
      The 3000-zip Content on my PC (B in the diagram) I just manually copy over to DropBox (F), and keep these up-to-date when I create new zips.
      Also, whenever I update my app code, I intend (this is probably the most painful bit) to manually copy the “\bin\release” folder contents from my PC (A) over to web host (E), so that the Updater app on user’s PCs can synchronise the executable and resource files with any newer versions I have created, without the user having to download a new version of the installer, and potentially uninstall/reinstall to go from v1 to v2 of the app.
      This above bit I think is the least possible, as I have found out that just by moving the \bin\release\ executable and resources from one location to another on my PC, it no longer works if I double-click the .exe file, it only works from the project’s original \bin\release folder  . Is this expected? In my naivety I was thinking that once the user had installed v1 of the application "properly" using ClickOnce (and therefore their PC was checked for .Net framework etc) I could then just have my Updater overwrite the .exe and other embedded resource files (like bitmaps) with updates and it’d just work. I guess this would have worked with VS2005, but not with VS2017 hey.
      So, after writing all that out, and thanks for reading this far, I guess that if it wasn’t for the 3000 zips, I could package all the above as ClickOnce, and let Microsoft manage the updating for me. I don’t fancy adding the 3000 zips to my project as Content [copy if newer] to enable this to happen, but I did fancy having a go at writing the standalone Updater package, which would synchronise the files between DropBox and the user’s PC, based on the xml manifest I create.
      Any feedback you may have would be greatly appreciated, as this community has proven invaluable to me so far

  • Advertisement
  • Advertisement

OpenGL Understanding data flow in a multi-threaded render pipeline

Recommended Posts

I'm having some difficulty understanding how data would flow or get inserted into a multi-threaded opengl renderer where there is a thread pool and a render thread and an update thread (possibly main). My understanding is that the threadpool will continually execute jobs, assemble these and when done send them off to be rendered where I can further sort these and achieve some cheap form of statelessness. I don't want anything overly complicated or too fine grained,  fibers,  job stealing etc. My end goal is to simply have my renderer isolated in its own thread and only concerned with drawing and swapping buffers. 

My questions are:

1. At what point in this pipeline are resources created?

Say I have a

class CCommandList
   void SetVertexBuffer(...);
   void SetIndexBuffer(...);
   void SetVertexShader(...);
   void SetPixelShader(...);

borrowed from an existing post here. I would need to generate a VAO at some point and call glGenBuffers etc especially if I start with an empty scene. If my context lives on another thread, how do I call these commands if the command list is only supposed to be a collection of state and what command to use. I don't think that the render thread should do this and somehow add a task to the queue or am I wrong?

Or could I do some variation where I do the loading in a thread with shared context and from there generate a command that has the handle to the resources needed.


2. How do I know all my jobs are done.

I'm working with C++, is this as simple as knowing how many objects there are in the scene, for every task that gets added increment a counter and when it matches aforementioned count I signal the renderer that the command list is ready? I was thinking a condition_variable or something would suffice to alert the renderthread that work is ready.


3. Does all work come from a singular queue that the thread pool constantly cycles over?

With the notion of jobs, we are basically sending the same work repeatedly right? Do all jobs need to be added to a single persistent queue to be submitted over and over again?


4. Are resources destroyed with commands?

Likewise with initializing and assuming #3 is correct, removing an item from the scene would mean removing it from the job queue, no? Would I need to send a onetime command to the renderer to cleanup?

Edited by getoutofmycar

Share this post

Link to post
Share on other sites

First off, welcome to GameDev.

A lot of your questions have super open ended answers and they all depend on how your engine works and how you designed it to work.

A good read about the flow of how a modern 3D engine works is found in these blog posts about the Autodesk Stingray Engine (now discontinued I believe): http://bitsquid.blogspot.com/2017/02/stingray-renderer-walkthrough.html

It's a general great blog, reading over older posts wouldn't hurt either.

Another good read, though it's mostly for Direct3D 11/12 and Vulkan (but the concepts are sound and would work for OpenGL) if I recall correctly is: 

As this sounds like it is your first go at a multithreaded engine, you're quite likely going to make design mistakes and likely refactor or restart several times, which is fine.  It's part of learning and the best way to learn is from ones own mistakes.

Share this post

Link to post
Share on other sites

Thanks for the welcome Mike! I agree they are open ended as you deduced. I have a very vague idea of what to do but new to the threading bits so wouldn't mind blindly following anyone's working approach until it sits with me and I can experiment. I will give those articles a read, I've also read the excellent high level ones over on the molecular blog but the inner workings are lost on me. 

Share this post

Link to post
Share on other sites

Thanks. I've actually watched these and more, Naughty Dog's one and others as well, but they are all still high level overviews that eventually wind down to stuff about atomics, actors, how to scale and none really delve into the intrinsics that I ask about but will no doubt be useful once I can get this running properly. Stuff like bgfx seems like what I would want but is a bit too dense to pick apart with all the backend stuff and whatnot.

Maybe my skull is extra thick, but none of my questions are answered in these. As I said before I do have a general idea of what to do, I'm more looking for actual implementation details mostly around how to handle resources, even pseudocode would be fine. I'm still reading the articles you linked above and will finish them tonight, hopefully I can glean anything from them to translate to code.

Edited by getoutofmycar

Share this post

Link to post
Share on other sites

The easiest way to get started is with a single "main thread" and an unstructured job system. The API for that might look as simple as:

void PushJob( std::function<void()> );

From your main thread, whenever you have a data-parallel workload, you can then farm it off to some worker threads:

vector<Object*> visible;
for( int i=0; i!=objects.size(); ++i )
  if( IsVisible(camera, objects[i]) )

//jobified for a split into 4 jobs:
#define NUM_THREADS 4
vector<Object*> visiblePerThread[NUM_THREADS];
Atomic32 jobsComplete[NUM_THREADS] = {0};
for( int t=0; t!=NUM_THREADS; ++t )
  vector<Object*>& visible = visiblePerThread[t];
  Atomic32& jobComplete = jobsComplete[t];
  int minimumWorkPerJob = 64;//dont bother splitting up workloads smaller than some amount
  int workPerThread = max( minimumWorkPerJob, numObjects/NUM_THREADS );
  //calculate a range of the data set for each thread to consume
  int start = workPerThread * t;
  int end = workPerThread * (t+1);
  start = min(start,numObjects);
  end = min(end,numObjects);
  if( start == end )//if nothing for this thread to do, just mark as complete instead of launching
    jobComplete = 1;
  else//push this functor into the job queue
    PushJob([&objects, &camera, &visible, start, end]()
      for( int i=start; i!=end; ++i )
        if( IsVisible(camera, objects[i]) )
      jobComplete = 1;
//at some point before "visible" is to be used:
//use one thread to join all the results into a single list
for( int t=1; t!=NUM_THREADS; ++t )
  //block until the job is complete
  BusyWaitUntil( [&](){ jobsComplete[t] == 1; } );
  //append result set [i] onto result set [0]
  visiblePerThread[0].insert(visiblePerThread[0].end(), visiblePerThread[i].begin(), visiblePerThread[i].end());
vector<Object*>& visible = visiblePerThread[0];

This simple job API is easy to use from anywhere, but adds some extra strain to its users. e.g. above, the user of the Job API needs to (re)invent their own way of figuring out that a job has finished each time. In this example, the user makes an atomic integer for each job, which gets set to 1 when the job is finished. The main thread can then busy wait (very bad!) until these integers change from 0 to 1.

In a fancier job system, PushJob would return some kind of handle, which the main thread could pass into a "WaitUntilJobIsComplete" type function.

This is the basics of how game engines spread their workloads across any number of threads these days. Once you're comfortable with these basic job systems, the very fancy ones use pre-declared jobs structures and pre-scheduled graphs, rather than the on-demand, ad-hoc "push anything, anytime" structure, above.

The other paradigm is having multiple "main" threads -- e.g. a game update thread and a rendering thread. This is basically just basic message passing with one very big message -- the game state required by the renderer.

Going off this bare-bones/basic job system, to answer your questions --
1. Probably on the main thread, maybe on a job if it's parallelisable work.
2. Above I used simple atomic flags. If you're using a nice threading API, then semaphores might be a safer choice.
3. Yes.
4. Objects are not persistent in the job queue -- the main thread pushes transient algorithms into the job queue, so there's nothing to remove. The logic is the same as a single-threaded game.

With a fancier job system, the answers could/would be different :)

17 hours ago, getoutofmycar said:

I would need to generate a VAO at some point and call glGenBuffers etc especially if I start with an empty scene.

GL is the worst API for this. In D3D11/D3D12/Vulkan, resource creation is free-threaded (creation of textures/buffers/shaders/states can be done from any thread), and in D3D12/Vulkan you can also use many threads to record actual state-setting/drawing commands (D3D11 can too, but you get no performance boost from it, generally). It's probably worthwhile to do all your your GL calls on a single "main rendering thread", rather than trying to make your own multi-threaded-command-queue wrapper over GL.

Nonetheless, the entire non-GL / non-D3D part of your renderer can still be multi-threaded. That involves preparing drawable objects, traversing the scene, culling/visibility, sorting objects, etc.... In a D3D12/Vulkan version, you can also multi-thread the actual submission of drawable objects to the API.

Share this post

Link to post
Share on other sites

Thank you, this is some great affirmation on my shaky logic. Your talk was what actually really inspired me to try my hand at this being the most approachable for a threading newbie. I think one of my misconceptions was about how I would be blocking the rendering thread when doing any allocation. Going to finish up my simple spec and tackle this over the weekend.

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Advertisement