I need option 3 in what you mention.

My problem is not only the rendering, but also all the GUI.

So I have:

Game logic thread: runs the game and modifies the resources.

GUI thread: handles the GUI events and also renders the views.

I want the game logic to be compilable in headless mode, and optionally have a plugin with the GUI and rendering. Think of it as a simulation of some physical process:

a) we can have the simulation configured and modified via a GUI, and also visualized. This situation requires the headless mode game logic executable, and the GUI/rendering plugin loaded

b) But other times, we simply want to be able to run, say, 10 simulations in parallel, in headless mode: in that case, we are only interested in the data the simulation produces. This situation only requires the headless mode game logic executable

So all the dependencies to the GUI and OpenGl should be in the plugin, so that the game logic can also easily run on machines that do not have any display (i.e. no OpenGl or graphic card installed)

So I agree that duplicating the resources is a tremendous amount of work, and many things can go wrong if not done properly. So, let's for now forget the resource duplication, and consider that we have only one thread doing everything.

What I currently have is following:

class CObject_A
{ 
    CObject_A(); 
    virtual ~CObject_A(); 

    void doCalculationAndModifyObject(); 
    void renderObject() 

    CData* data; 
};

I basically currently have everything in one class (actually many classes, but for simplification sake). Because of that I cannot compile the gamelogic in headless mode (above class has the dependencies to the GUI/OpenGl). I need to separate the above class in 2 separately compilable entities, with least effort.

For that reason I mentioned the possibility to simply cast a class to another class, if the member variables are the same.

So I want to be able to do following:

// gameLogic executable:
CObject_A* object=new CObject_A();
object->doCalculationAndModifyObject();
callPluginRoutine(object);

And in my plugin:
void callPluginRoutine(CObject_A* obj)
{
    obj->renderObject();
}

If I want to be able to compile my gameLogic executable in headless mode, I am not allowed to have routine "renderObject()" in class CObject_A during compilation.

Phil, perfectly agree with you. If I had to rewrite the whole project, that's how I would do it. Unfortunately I have to modify an existing code (written by another person) that has several hundreds of thousands lines. It would take several months to do that. On the other hand, if I have a solution (maybe not that elegant) that works, and that can slowly be refactored while keeping current working solution, it would be great.

With the initial idea of simply casting an object of class A to and object of class B (where the two classes are identical except for the member functions), I could put a few ifdefs in the class definitions, I could compile class A in item A, and class B in item B, but using the common, existing class AB.

@floatingwoods still I'm not sure if I understand what you are trying to achieve but from what I see, you need exactly what @phil_t described ( correct me if I'm wrong, but this looks to me like typical MVC pattern ). From your description you need two interfaces to the same set of data. There is no real reason why those interfaces must duplicate the data.

With the initial idea of simply casting an object of class A to and object of class B (where the two classes are identical except for the member functions), I could put a few ifdefs in the class definitions, I could compile class A in item A, and class B in item B, but using the common, existing class AB.

This will work only if you don't use virtual functions and maintain the structural integrity, but I don't see why you can't create classes A and B for which a memebr of class C ( probably pointer ) would point at the shared data? Also I'm not sure what you call as a plugin? Are you planning to compile part of the code as shared library or something?

I think you're taking wrong direction here. Why in the headless mode simply render() function would not do anything? This is pretty common practice in game engines where they have to do some work without really rendering anything. You link against plugin library which either supports rendering or not. I did something similar long time ago with one of my old engines to be able to run only in serialisation mode. So it was compiled against "dummy" renderer. If you want more flexibility, place your plugin in the shared library and link it in the runtime by opening it and grabbing symbols. In that case you could determine if you support rendering but simply checking if the symbol exists.

In your idea you are sacrificing the binary compatibility and make it harder to maintain. But as I say, I'm not sure what you're really trying to achieve :) If I wanted to be able to process/render my data with plugin, then I'd go with proper interface through which I can access my data and move functionality to the library ( shared if possible ). Both, plugin and main application would need to be compiled against common header that defines data structure, so if you create ObjectA in main application and then you give ObjectA to the plugin function, the plugin will be aware of the data structure. Also not everything has to be class. Consider that member functions are in fact normal functions which take pointer to the 'this' object as first argument ( in a simple view of things ;) ). Which means, instead of creating ClassA::UpdateSimulation(), ClassB::Render() you would create for example in your plugin code:

#include <your-common-data-header.h>

extern "C" void RenderImpl( Data* pData )
{
  // do render your data here or leave empty
}

Your main application would look for a function pointer from the plugin and call it if it exists or ignore. It's just very simple example of decoupling plugin and main application. But as I said, I'm not sure if I understand what you're trying to do ;)

Edit: I forgot to mention that the plugin could be also responsible for spawning own threads if necessary, so when you run application in headless mode, there are no needless threads floating around. This is how I implemented OpenGL backend for my latest project, which requires to have a main thread for the command queue, while Vulkan plugin doesn't need to create one as command queue there works hmm... "out of the box" ;) But interface between an application and Vulkan, OpenGL and DX12 is exactly the same, they are just different in terms of the way they do things inside the shared library. For debugging I also have a dummy plugin ( which is used when running test cases on build machine - so this is sort of headless mode ;) ).

Well, all this makes perfect sense. It seems I got scared (and mixed-up my thoughts) because of the thread, separate entities (exe + dll) and existing code aspect.

The fact that I need different threads is that my game logic already works using several dozens of threads (but perfectly synchronized). So it only makes sense to have a separate thread for the GUI and rendering (e.g. Qt also requires to run the main Qt code via the application main thread)