Jump to content

  • Log In with Google      Sign In   
  • Create Account


Member Since 02 Oct 2013
Offline Last Active Oct 25 2016 10:40 AM

Posts I've Made

In Topic: Python C API: Undefined functions in module

25 October 2016 - 10:11 AM

The API changed, in 3.x the function PyEval_EvalCode() takes a PyObject* instead of a PyCodeObject*. Probably because someone decided that maybe code objects aren't the only type of object that can be evaluated? Who knows. Py_CompileString() will return a PyCodeObject* so you're safe to cast it.

I was able to solve it with some help from the #python IRC channel: Apparently, passing the "globals" dict from the __main__ module to PyEval_EvalCode() will cause whatever functions are in the code object to be registered globally to the __main__ module, but will still be unavailable to the local module. The documentation is extremely sparse and I have to admit I don't understand why the following code works. What exactly happens with "globals" and "locals" in the call to PyEval_EvalCode(compiled, globals, locals)?


Anyway, here's the code that works:

void ScriptInstance::reload()
{    /*
     * Need to clear everything that is currently in the module's dictionary,
     * so it's not possible to accidentally call old functions that no longer
     * exist.
     * In order for the builtin functions to be defined, copy the __main__
     * module's dictionary into the new module's local dictionary
    PyObject* locals = PyModule_GetDict(module_);
    PyObject* main = PyImport_AddModule("__main__");
    PyObject* globals = PyModule_GetDict(main);
    PyModule_AddStringConstant(module_, "__file__", "");
    PyDict_Merge(locals, globals, 0);

     * Evaluate the code into this module's local dictionary.
    PythonObject<> eval = PyEval_EvalCode(compiled.get(), locals, locals);

    flushFunction_ = PyObject_GetAttrString(module_, "flush");


Upon reloading, I have to clear the module's local dictionary of all items, to make sure it's impossible to call functions that are no longer defined in the script.


This isn't enough, because now the dictionary is empty, which means any scripts that are evaluated using this dictionary won't have access to any built-in commands like "print()" or "import". To fix this, I get the dictionary from __main__ (which defines all of these things) and merge it with the local dictionary.


Once that's done, I pass "local" twice to PyEval_EvalCode(). That fixes it.


Like I said: I don't fully understand why this works and would appreciate some explanation.

In Topic: Reloading Python script (C API)

24 October 2016 - 06:17 AM

Thanks for the suggestions, this is exactly what I ended up doing. In case some unfortunate soul in the future has a similar problem to mine, here is exactly how I solved reloading scripts.


A quick disclaimer: This unfortunately only works for the functions you explicitly want to support for reloading. If your script imports other scripts, you're pretty much doomed with this approach. You should instead try to reset the interpreter.


I should also note that using a sub-interpreter (Py_NewInterpreter() and PyThreadState_Swap()) for every script you load won't solve this issue. There's a function in the python source code (see import.c) that maintains a global list of imported modules across all instances. There is no way to remove yourself from this list after your module is loaded.


In my situation, I wanted to expose two C++ callback functions to python. This is the class definition:

class ScriptInstance : public common::RefCounted
    ScriptInstance(const QString& fileName);

    void reload();

    void flush();
    void apply(ubox::Frame* frame);

    QString fileName_;
    previewer::PythonObject<> module_;
    previewer::PythonObject<> flushFunction_;
    previewer::PythonObject<> applyFunction_;

Some notes: PythonObject is a wrapper around PyObject*. It steals a reference on construction and calls Py_XDECREF() on destruction. flush() and apply() are the two functions I want to expose to python.


The first thing I do (before creating any ScriptInstance instances) is intialise the python interpreter:

bool PythonInterpreter::initialise()
        return true;

    // Add built-in ubox module before initialisation
    PyImport_AppendInittab("ubox", PyInit_ubox);

    static wchar_t programName[] = L"my app";
        defaultLogger.logError("Failed to initialise python interpreter");
        return false;

    // TODO this assumes the default location of the python interpreter

    // Import modules
    PyObject* tmp;
    tmp = PyImport_ImportModule("ubox"); // XXX This returns NULL for some reason? It loads nevertheless...
    tmp = PyImport_ImportModule("numpy");
    tmp = PyImport_ImportModule("scipy");

    return true;

Then, when I load a script, I create a new ScriptInstance and pass in the file name. In its constructor I create a new python module and register it under the name of the script (without the .py extension).

ScriptInstance::ScriptInstance(const QString& fileName) :
    // Need the global python interpreter to exist
    if(PythonInterpreter::initialise() == false)

     * Create a python module for this script instance, in which functions and
     * other data can be registered.
    module_ = PyModule_New(fileInfo.fileName().replace(".py", "").toStdString().c_str());
    PyModule_AddStringConstant(module_, "__file__", "");



The reload() function is responsible for loading the file, compiling it, and extracting the relevant functions flush() and apply():

void ScriptInstance::reload()
    QFile file(fileName_);
    QByteArray bytes = file.readAll();

    PythonObject<> compiled = Py_CompileString(bytes.data(), fileName_.toLatin1().data(), Py_file_input);
    // error handling omitted

    // these are all borrowed references
    PyObject* main = PyImport_AddModule("__main__");
    PyObject* globals = PyModule_GetDict(main);
    PyObject* locals = PyModule_GetDict(module_);

    PythonObject<> eval = PyEval_EvalCode(compiled, globals, locals);
    // error handling omitted

     * Get the two python functions flush() and apply() and make sure they're
     * correct and can be used.
    flushFunction_ = PyObject_GetAttrString(module_, "flush");
    applyFunction_ = PyObject_GetAttrString(module_, "apply");
    // error handling omitted

PyImport_AddModule("__main__") will return a borrowed reference to the __main__ module, which in turn contains a dictionary that contains all of the imported modules and functions thus far. This dictionary must be passed to PyEval_EvalCode() when you're evaluating your script, otherwise basic functions such as print() won't be recognised.


The locals dictionary is retrieved from the module that was previously created in the constructor and it will hold all of the data that is local to our module (i.e. it will store the flush() and apply() function objects).


After evaluating the code with the proper dictionary objects, you will be able to retrieve the function objects from the module with PyObject_GetAttrString(module_, "func").

In Topic: Reloading Python script (C API)

23 October 2016 - 08:33 AM

Turns out restarting the interpreter segfaults somewhere deep in numpy (multiarray/refcount.c)


I'll submit a bug report but the lack of ability to reload scripts in python is really annoying.

In Topic: Reloading Python script (C API)

23 October 2016 - 08:21 AM

I understand the problems of reloading, but this application really is a special case. I can guarantee that all references will be destroyed at the point where the module gets reloaded.


I suppose I could also just restart the entire interpreter...

In Topic: C++ : Extend class in dll

17 October 2016 - 12:49 PM

Ah, I should also mention that the code I posted is mostly taken from the Urho3D project, which is heavily based on this templated factory paradigm. You should check out their code under Source/Urho3D/Core/Context.cpp