Jump to content

  • Log In with Google      Sign In   
  • Create Account


Member Since 02 Oct 2013
Offline Last Active Today, 04:10 PM

Posts I've Made

In Topic: Reloading Python script (C API)

Today, 06:17 AM

Thanks for the suggestions, this is exactly what I ended up doing. In case some unfortunate soul in the future has a similar problem to mine, here is exactly how I solved reloading scripts.


A quick disclaimer: This unfortunately only works for the functions you explicitly want to support for reloading. If your script imports other scripts, you're pretty much doomed with this approach. You should instead try to reset the interpreter.


I should also note that using a sub-interpreter (Py_NewInterpreter() and PyThreadState_Swap()) for every script you load won't solve this issue. There's a function in the python source code (see import.c) that maintains a global list of imported modules across all instances. There is no way to remove yourself from this list after your module is loaded.


In my situation, I wanted to expose two C++ callback functions to python. This is the class definition:

class ScriptInstance : public common::RefCounted
    ScriptInstance(const QString& fileName);

    void reload();

    void flush();
    void apply(ubox::Frame* frame);

    QString fileName_;
    previewer::PythonObject<> module_;
    previewer::PythonObject<> flushFunction_;
    previewer::PythonObject<> applyFunction_;

Some notes: PythonObject is a wrapper around PyObject*. It steals a reference on construction and calls Py_XDECREF() on destruction. flush() and apply() are the two functions I want to expose to python.


The first thing I do (before creating any ScriptInstance instances) is intialise the python interpreter:

bool PythonInterpreter::initialise()
        return true;

    // Add built-in ubox module before initialisation
    PyImport_AppendInittab("ubox", PyInit_ubox);

    static wchar_t programName[] = L"my app";
        defaultLogger.logError("Failed to initialise python interpreter");
        return false;

    // TODO this assumes the default location of the python interpreter

    // Import modules
    PyObject* tmp;
    tmp = PyImport_ImportModule("ubox"); // XXX This returns NULL for some reason? It loads nevertheless...
    tmp = PyImport_ImportModule("numpy");
    tmp = PyImport_ImportModule("scipy");

    return true;

Then, when I load a script, I create a new ScriptInstance and pass in the file name. In its constructor I create a new python module and register it under the name of the script (without the .py extension).

ScriptInstance::ScriptInstance(const QString& fileName) :
    // Need the global python interpreter to exist
    if(PythonInterpreter::initialise() == false)

     * Create a python module for this script instance, in which functions and
     * other data can be registered.
    module_ = PyModule_New(fileInfo.fileName().replace(".py", "").toStdString().c_str());
    PyModule_AddStringConstant(module_, "__file__", "");



The reload() function is responsible for loading the file, compiling it, and extracting the relevant functions flush() and apply():

void ScriptInstance::reload()
    QFile file(fileName_);
    QByteArray bytes = file.readAll();

    PythonObject<> compiled = Py_CompileString(bytes.data(), fileName_.toLatin1().data(), Py_file_input);
    // error handling omitted

    // these are all borrowed references
    PyObject* main = PyImport_AddModule("__main__");
    PyObject* globals = PyModule_GetDict(main);
    PyObject* locals = PyModule_GetDict(module_);

    PythonObject<> eval = PyEval_EvalCode(compiled, globals, locals);
    // error handling omitted

     * Get the two python functions flush() and apply() and make sure they're
     * correct and can be used.
    flushFunction_ = PyObject_GetAttrString(module_, "flush");
    applyFunction_ = PyObject_GetAttrString(module_, "apply");
    // error handling omitted

PyImport_AddModule("__main__") will return a borrowed reference to the __main__ module, which in turn contains a dictionary that contains all of the imported modules and functions thus far. This dictionary must be passed to PyEval_EvalCode() when you're evaluating your script, otherwise basic functions such as print() won't be recognised.


The locals dictionary is retrieved from the module that was previously created in the constructor and it will hold all of the data that is local to our module (i.e. it will store the flush() and apply() function objects).


After evaluating the code with the proper dictionary objects, you will be able to retrieve the function objects from the module with PyObject_GetAttrString(module_, "func").

In Topic: Reloading Python script (C API)

Yesterday, 08:33 AM

Turns out restarting the interpreter segfaults somewhere deep in numpy (multiarray/refcount.c)


I'll submit a bug report but the lack of ability to reload scripts in python is really annoying.

In Topic: Reloading Python script (C API)

Yesterday, 08:21 AM

I understand the problems of reloading, but this application really is a special case. I can guarantee that all references will be destroyed at the point where the module gets reloaded.


I suppose I could also just restart the entire interpreter...

In Topic: C++ : Extend class in dll

17 October 2016 - 12:49 PM

Ah, I should also mention that the code I posted is mostly taken from the Urho3D project, which is heavily based on this templated factory paradigm. You should check out their code under Source/Urho3D/Core/Context.cpp

In Topic: C++ : Extend class in dll

17 October 2016 - 12:39 PM

Typically this is achieved through some form of dynamic type information, be it using templates + typeid() or some other form of unique type identification


The basic idea consists of two classes. A "Manager" and a "Factory". The manager stores a collection of factories and associated type identification, and the factories know how to instantiate the type you need.


The following is a complete example using std::type_info. Note that you will need C++11 because of hash_code(). If you must use c++03 then you can replace hash_code() with name() and run that through your own hash function (or change the type in the container from size_t to std::string).

#include <typeinfo>
#include <map>
#include <string>
#include <iostream>

class Factory {
    virtual void* Create() = 0;

template <class T>
class FactoryImpl : public Factory
    virtual void* Create() override
        return new T();

class Manager
    template <class T>
    void RegisterObjectFactory()
        factoryMap_[typeid(T).hash_code()] = new FactoryImpl<T>();

    template <class T>
    T* CreateObject() const
        auto factory = factoryMap_.find(typeid(T).hash_code());
        if(factory == factoryMap_.end())
            return NULL;
        return static_cast<T*>(factory->second->Create());

    template <class T>
    void DestroyObject(T* object)
        delete object;

    std::map<size_t, Factory*> factoryMap_;

// ----------------------------------------------------------------------------
// Example usage
// ----------------------------------------------------------------------------
class Test
    void Greet() { std::cout << "it works!" << std::endl; }

int main()
    Manager manager;
    Test* test = manager.CreateObject<Test>();



One pitfall to look out for with factories is to make sure you delete the object in the same DLL you created it. This is why Manager::DestroyObject() is necessary. If you were to convert it from raw pointers to std::shared_ptr then this would no longer be necessary to do.


With this method you would need to pass the manager instance to your plugin DLLs when they start (typically achieved by defining your own start_plugin() function). Then your plugins can call Manager::RegisterObjectFactory<Whatever>() to register the types they require. This will enable the core application to start instantiating types that weren't originally there when compiling.


You might consider extending the example to make it possible to pass the actual class name to the manager, and not just the has code. For example:


instead of


For this you would need to store typeid(T).name() in the corresponding factory class.