Graphics programming language

Graphics and GPU Programming Programming OpenGL

Started by Shnoutz February 08, 2017 08:00 PM

10 comments, last by Shnoutz 7 years, 1 month ago

Shnoutz

356

Author

February 08, 2017 08:00 PM

Hi,

I am a professional graphics programmer and I create scripting/programming languages as an hobby. I got this idea that I wanted to share with you.

I am reading and learning about low level graphics APIs and the reason why they exist. In DirectX11/OpenGL a lot of the GPU work, like resource barriers for example, is hidden and executed by the driver.

Now, because the driver doesn't know what your frame looks like it has to execute the worst case scenario and execute more barriers that may be required.

(I think DX11 drivers now are quite clever and do prediction to reduce that problem but you get my point)

DX12/Vulkan somewhat solves this issue by letting the programmer decide where to execute the barriers by exposing them as an API concept.

That is a major plus but it is very error prone and if not done correctly can lead to major performance issues.

Now this got me thinking... What if we created a programming language that allowed to define explicitly what a full frame looks like. The steps and the resources involved in those steps.

We could then look at theses steps and figure out exactly where to put the barriers. Re-order the steps for optimal performance. We could also look at what are the dependencies between steps and probably figure out a way to automatically dispatch the work on different queues (copy/dma, compute & graphics).

I have the feeling that with new low level APIs this door is now opened. Static analysis and optimization (of full frames)... Something every compilers do for CPU code. Why not GPU code?

Any thoughts on that?

Gab.

Infinisearch

3,058

February 08, 2017 08:43 PM

I suggested something similar here: https://forum.beyond3d.com/threads/driver-optimizations-and-the-api.56087/

Also doesn't vulkan have the renderpass which is a partial implementation of the idea?

But I agree with you in that analysis of a frame would be possible if it were outlined and that could be used to optimize gpu execution.

-potential energy is easily made kinetic-

JoeJ

4,187

February 08, 2017 09:23 PM

That's a nice idea and i always think about some framework that would at least simplify things a bit, makes dependencies more clear, less code in general...

It is questionable how far this can go while avoiding to loose performance for simplicity again.
Think of a compute shader that writes only to a small region of a large buffer - a barrier over the entire buffer would be a waste, we would need to specify things with more details if 'auto' mode might miss something.

But yes, we need something like that. If this would work for various hardware and APIs it could even turn hobby to business.

MJP

20,295

February 09, 2017 08:57 AM

I'm not sure that you really need a full programming language to do this, although it might help for some cases. A few people have already been experimenting with building task-graph-esque structures and walking them to find points where barriers are needed. I believe that Yuriy O'Donnell is going to present something along these lines at GDC.

The Blog | The Book

Shnoutz

356

Author

February 09, 2017 09:45 PM

I was thinking of a programming language because it is essentially a compact way to express a tree of resources and operations.
I also like the idea of generating code rather than evaluating a tree like structure at run-time.

(I do convert a lot of my code from a script like language to c++)

Too bad I wont be attending GDC this year I would have loved to see that presentation.

JoeJ

4,187

February 10, 2017 06:44 AM

I also like the idea of generating code rather than evaluating a tree like structure at run-time.

I'd say we need a variable graph to handle various hardware configurations but more important various graphics options.
For instance if the user increases a detail setting, doing things async can become a loss and it's better to serialize them. It also matters here how large the GPU is.
Script -> C++ can't give that flexibility, you need at least some kind of interpreter or do it just with C++ directly.

... and there we are close where we started. I always end up keep using the API directly. It is hard to reduce the complexity of something if it is simply necessary.

Hodgman

52,717

February 10, 2017 07:20 AM

I'd say we need a variable graph to handle various hardware configurations but more important various graphics options. For instance if the user increases a detail setting, doing things async can become a loss and it's better to serialize them. It also matters here how large the GPU is. Script -> C++ can't give that flexibility, you need at least some kind of interpreter or do it just with C++ directly.

You could always produce many permutations of C++ code. Instead of branching at a fine grained level inside the C++ level (e.g. if shadow level is low, call DrawLowShadows) you could just have one C++ entry point for each permutation of the graph.

e.g. have this language generate a bunch of C++ functions like:

void GeneratedGraph_ShadowsLow_MemoryUnder1024_ShadersHigh_LightingMedium(blah)

. 22 Racing Series .

JoeJ

4,187

February 10, 2017 08:56 AM

Yep,

Script -> C++ can't give that flexibility

and of course a script CAN generate C++ with branches if permutation count explodes :)

JorenJoestar

381

February 10, 2017 03:43 PM

In my opinion it really depends what you want to express with the language.

If you want to describe a rendering pipeline (that FINALLY with Vulkan/Dx12 is stateless) then it's one thing.
If you want to describe different rendering configuration, is another.
If you want a api-agnostic way of issuing drawing command, then is another thing.

For the full frame definition...you need to express the dependencies of a frame.

Buffer/Textures are the main ones.

From a drawing/compute point of view, you are writing into a buffer.
Who needs that buffer then dictates the order.

This is an old concept but it could be a starting point!
There is also someone that did some work on that:

https://www.gamedev.net/blog/1930/entry-2262315-designing-a-high-level-render-pipeline-part-3-a-visual-interface/

I personally use this separation since quite a while (explicit render pipelines, stages and their dependency), and you can describe everything more easily.

Vulkan finally has the RenderPass for it!

Also, into writing a custom language, I suggest you to look into Antlr (version 4):

It's a library to create a lexer and a parser from a grammar:

http://www.antlr.org/

https://github.com/antlr/grammars-v4

Version 4 is pretty solid and they improved a lot the expressiveness of their language.

So for example, with your language you can generate c++ code quite easily once you have a tree of resources you need to draw!

---------------------------------------http://badfoolprototype.blogspot.com/

Shnoutz

356

Author

February 10, 2017 08:44 PM

I think the first step would be to define the components of the language as generic as possible.

What the most basic operations that are performed in a frame? Clear, copy, dispatch, draw (and present maybe).

These operations have inputs and outputs, figuring out dependencies is quite easy.

Granted the shader code is required to figure out if a texture is used as a UAV or as a SRV but really, adding a pseudo shading language is not too scary I've done something similar in the past

(maybe not the first thing I'll do ;) ). Still that would not be enough to know if, for example, a buffer was fully or partially written to but there might be a way to express sub resource regions.

I'm starting to think that the basic operations are like instructions and the shader code is like micro-instructions.

With "instructions" and dependencies we can do lots of cool stuff like re-ordering and eliminating duplication.

I have started to work on a small prototype that takes code as input and spit out a graph with dependencies as output... I think its the first step.

But its easy to imagine where that could lead... Automatic barriers, operation reordering, automatic async compute (given a descriptions of the hardware queues), automatic volatile resource allocation/aliasing, descriptor management.

That's a job for a full team of professional and for quite a while but ill try just for fun.

Graphics programming language

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Graphics programming language

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines