Sign in to follow this  

Reloading Python script (C API)

This topic is 420 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I am developing an application written in C++/Qt that embeds the python interpreter and exposes some functionality through python bindings. I'm using CPython API to do this (mostly because I wanted to learn more about how to do this).

 

I want to add "hot reloading" of python scripts to this application. When the script is modified and saved to disk, I want to reload that script and re-execute it. I feel like I'm really close to the solution but I can't quite get it to work.

 

This is the function I use to load (and reload) my python script:

http://pastebin.com/1PDE7EY1

 

This works fine the first time I call it, but when I call the function a second time (and pass the same argument, namely the path to my test.py script), I get the error: ImportError("bad magic number in 'test').

 

I have also tried using PyImport_ReloadModule() instead of trying to delete the entry in sys.modules, but this also causes the same error.

 

What is really interesting is that if I place a breakpoint on PyImport_Import() and then resume execution it works! Why??

 

Now, I've been told that it's "really hard or impossible" to reload python scripts, especially when they start importing other scripts and are referenced in other scripts. HOWEVER, it works with the breakpoint "hack" and I can also get it to work just fine using the interactive console:

 

This is the content of test.py:

def test():
    print("one!")

I load and execute it in the python console:

>>> import test
>>> test.test()
one!

I now edit the script (without closing the python console) to this:

def test():
    print("two!")

In order for the change to take effect, I can do this:

>>> import sys
>>> del sys.modules['test']
>>> del test
>>> import test
>>> test.test()
two!

All I want to do is reproduce this using the C API and I can't figure out how my C++ code differs from the interactive session.

Edited by TheComet

Share this post


Link to post
Share on other sites

Now, I've been told that it's "really hard or impossible" to reload python scripts, especially when they start importing other scripts and are referenced in other scripts. HOWEVER, it works with the breakpoint "hack" and I can also get it to work just fine using the interactive console:

 

The reason why it's hard, is because of references to old data.

>>> import test
>>> test.test()
one!
>>> f = test.test

# edit test.py

>>> del test
>>> import sys
>>> del sys.modules['test']
>>> import test
>>> test.test()
two!
>>> f()
one!

Despite deletion of the test module, "f" hangs on to the old test function, so now you have 2 test.test functions.

Imagine what happens if you do this with a class, that you use to make objects, then replace the class, and make more objects.

 

Do you really want to end up with objects that come from two different classes with the same name, and potentially different behavior?

Share this post


Link to post
Share on other sites

Random idea:

 

You can just replace functions in Python by assigning a new implementation to it. So you don't need a reload, if you can enter the functions, and assign them to existing function names.

Last time I coded Python C API, there was a function to compile a piece of Python code. Maybe it even exists at Python level. Would it be feasible to give the new Python code as string, compile it, and assign the result to existing function names?

Share this post


Link to post
Share on other sites

Thanks for the suggestions, this is exactly what I ended up doing. In case some unfortunate soul in the future has a similar problem to mine, here is exactly how I solved reloading scripts.

 

A quick disclaimer: This unfortunately only works for the functions you explicitly want to support for reloading. If your script imports other scripts, you're pretty much doomed with this approach. You should instead try to reset the interpreter.

 

I should also note that using a sub-interpreter (Py_NewInterpreter() and PyThreadState_Swap()) for every script you load won't solve this issue. There's a function in the python source code (see import.c) that maintains a global list of imported modules across all instances. There is no way to remove yourself from this list after your module is loaded.

 

In my situation, I wanted to expose two C++ callback functions to python. This is the class definition:

class ScriptInstance : public common::RefCounted
{
public:
    ScriptInstance(const QString& fileName);


    void reload();


    void flush();
    void apply(ubox::Frame* frame);


private:
    QString fileName_;
    previewer::PythonObject<> module_;
    previewer::PythonObject<> flushFunction_;
    previewer::PythonObject<> applyFunction_;
};

Some notes: PythonObject is a wrapper around PyObject*. It steals a reference on construction and calls Py_XDECREF() on destruction. flush() and apply() are the two functions I want to expose to python.

 

The first thing I do (before creating any ScriptInstance instances) is intialise the python interpreter:

bool PythonInterpreter::initialise()
{
    if(Py_IsInitialized())
        return true;

    // Add built-in ubox module before initialisation
    PyImport_AppendInittab("ubox", PyInit_ubox);


    static wchar_t programName[] = L"my app";
    Py_SetProgramName(programName);
    Py_Initialize();
    if(!Py_IsInitialized())
    {
        defaultLogger.logError("Failed to initialise python interpreter");
        return false;
    }


    // TODO this assumes the default location of the python interpreter
    PySys_SetPath(
        L":"
        L":python/lib/python3.5"
        L":python/lib/python3.5/site-packages"
        L":python/lib/python3.5/lib-dynload"
    );


    // Import modules
    PyObject* tmp;
    tmp = PyImport_ImportModule("ubox"); // XXX This returns NULL for some reason? It loads nevertheless...
    tmp = PyImport_ImportModule("numpy");
    tmp = PyImport_ImportModule("scipy");


    return true;
}

Then, when I load a script, I create a new ScriptInstance and pass in the file name. In its constructor I create a new python module and register it under the name of the script (without the .py extension).

ScriptInstance::ScriptInstance(const QString& fileName) :
    fileName_(fileName)
{
    // Need the global python interpreter to exist
    if(PythonInterpreter::initialise() == false)
        return;

    /*
     * Create a python module for this script instance, in which functions and
     * other data can be registered.
     */
    module_ = PyModule_New(fileInfo.fileName().replace(".py", "").toStdString().c_str());
    PyModule_AddStringConstant(module_, "__file__", "");


    reload();
}

 

The reload() function is responsible for loading the file, compiling it, and extracting the relevant functions flush() and apply():

void ScriptInstance::reload()
{
    QFile file(fileName_);
    file.open(QIODevice::ReadOnly);
    QByteArray bytes = file.readAll();
    file.close();

    PythonObject<> compiled = Py_CompileString(bytes.data(), fileName_.toLatin1().data(), Py_file_input);
    // error handling omitted

    // these are all borrowed references
    PyObject* main = PyImport_AddModule("__main__");
    PyObject* globals = PyModule_GetDict(main);
    PyObject* locals = PyModule_GetDict(module_);

    PythonObject<> eval = PyEval_EvalCode(compiled, globals, locals);
    // error handling omitted

    /*
     * Get the two python functions flush() and apply() and make sure they're
     * correct and can be used.
     */
    flushFunction_ = PyObject_GetAttrString(module_, "flush");
    applyFunction_ = PyObject_GetAttrString(module_, "apply");
    // error handling omitted
}

PyImport_AddModule("__main__") will return a borrowed reference to the __main__ module, which in turn contains a dictionary that contains all of the imported modules and functions thus far. This dictionary must be passed to PyEval_EvalCode() when you're evaluating your script, otherwise basic functions such as print() won't be recognised.

 

The locals dictionary is retrieved from the module that was previously created in the constructor and it will hold all of the data that is local to our module (i.e. it will store the flush() and apply() function objects).

 

After evaluating the code with the proper dictionary objects, you will be able to retrieve the function objects from the module with PyObject_GetAttrString(module_, "func").

Share this post


Link to post
Share on other sites

This topic is 420 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this