Sign in to follow this  
kloffy

Boost Python - Calling a member function that was overridden in Python from C++

Recommended Posts

I'm trying to: 1) Define a base class in C++ with virtual member functions 2) Expose it to Python using boost python 3) Create a derived class in Python that overrides the virtual functions 4) Create an instance of the derived class in Python 5) Pass the instance to C++ 6) Call a member function from C++ resulting in having the Python override called If I interpret the documentation correctly (Virtual Functions with Default Implementations) this should be possible. However, I seem to be doing something wrong, because in my implementation the overridden function is not called.
#include <iostream>
#include <string>

#include <boost/python.hpp>
#include <boost/smart_ptr.hpp>

namespace py = boost::python;

class Object
{
public:
	Object(int _id): id(_id) {}
	virtual ~Object() {}
	int getId() const { return id; }

	virtual int f()
	{
		std::cout << "Object" << std::endl;
		return 0;
	}
private:
	int id;
};

struct ObjectWrap: public Object, public py::wrapper<Object>
{
    int f()
    {
		if (py::override f = this->get_override("f")) return f();
        return Object::f();
    }

    int default_f() { return this->Object::f(); }
};

BOOST_PYTHON_MODULE(Core)
{
	py::class_<ObjectWrap, boost::shared_ptr<Object> >("Object", py::init<int>())
		.add_property("id", &Object::getId)
		.def("f", &Object::f, &ObjectWrap::default_f)
    ;
}

int main(int argc, const char* argv[])
{
	Py_Initialize();
	
	try 
	{
		initCore();

		py::object main_module = py::import("__main__");
		py::object main_namespace = main_module.attr("__dict__");
		py::object ignored = py::exec(
			"import Core\n"
			"class CustomObject(Core.Object):\n"
			"	def __init__(self,id):\n"
			"		Core.Object.__init__(self,id)\n"
			"	def f(self):\n"
			"		print \"CustomObject\"\n"
			"		return 0\n"
			"object = CustomObject(1337)\n",
			main_namespace, main_namespace
		);

		py::object object = main_namespace["object"];
		object.attr("f")(); // -> CustomObject

		boost::shared_ptr<Object> o = py::extract< boost::shared_ptr<Object> >(object);
		o->f(); // -> Object (should be CustomObject?)
	}
	catch(const py::error_already_set&)
	{
		PyErr_Print();
	}

	Py_Finalize();

	return 0;
}



Share this post


Link to post
Share on other sites
When you specify a HeldType parameter for boost::python::class_<T>, it must be be either T, derived from T or a Dereferencible type where pointee<T>::type is T or derived from T. You are trying to use boost::shared_ptr<Object> when T is ObjectWrap. Object doesn't derive from ObjectWrap, so your code is malformed. You probably want to use a boost::shared_ptr<ObjectWrap> as the HeldType. Also, you need to give ObjectWrap an int constructor.

Share this post


Link to post
Share on other sites
Ok, thanks for clearing that up. I was somewhat confused about the HeldType and the whole wrapper business just made it worse. Here's a version that works:


#include <iostream>
#include <string>

#include <boost/python.hpp>
#include <boost/smart_ptr.hpp>

namespace py = boost::python;

class Object
{
public:
Object(int _id): id(_id) {}
virtual ~Object() {}

int getId() const { return id; }

virtual int f()
{
std::cout << "Object" << std::endl;
return 0;
}
private:
int id;
};

struct ObjectWrap: public Object, public py::wrapper<ObjectWrap>
{
ObjectWrap(int _id): Object(_id) {}

int f()
{
if (py::override f = this->get_override("f")) return f();
return Object::f();
}

int default_f() { return this->Object::f(); }
};

BOOST_PYTHON_MODULE(Core)
{
py::class_<ObjectWrap, boost::shared_ptr<ObjectWrap> >("Object", py::init<int>())
.add_property("id", &Object::getId)
.def("f", &Object::f, &ObjectWrap::default_f)
;
}

int main(int argc, const char* argv[])
{
Py_Initialize();

try
{
initCore();

py::object main_module = py::import("__main__");
py::object main_namespace = main_module.attr("__dict__");
py::object ignored = py::exec(
"import Core\n"
"class CustomObject(Core.Object):\n"
" def __init__(self,id):\n"
" Core.Object.__init__(self,id)\n"
" def f(self):\n"
" print \"CustomObject\"\n"
" return 0\n"
"object = CustomObject(1337)\n",
main_namespace, main_namespace
);

py::object object = main_namespace["object"];
object.attr("f")();

boost::shared_ptr<ObjectWrap> o = py::extract< boost::shared_ptr<ObjectWrap> >(object);
o->f();
}
catch(const py::error_already_set&)
{
PyErr_Print();
}

Py_Finalize();

return 0;
}




Note that I had to derive ObjectWrap from
py::wrapper<ObjectWrap>
rather than
py::wrapper<Object>
in order to get it to work. I'm not quite sure why that is...

Share this post


Link to post
Share on other sites
It has to do with arcane details behind copy semantics. You could keep the original py::wrapper<Object> if you used boost::noncopyable with the class_. Ex:

py::class_<ObjectWrap,
boost::shared_ptr<ObjectWrap>,
boost::noncopyable >

Share this post


Link to post
Share on other sites
One more question though: Now, do I really need to define a conversion in order to expose an instance of the base class to Python?
Is there any other way meant for doing something like that (based on the previous code):


boost::shared_ptr<Object> o(new Object(1337));

py::object main_module = py::import("__main__");
py::object main_namespace = main_module.attr("__dict__");
main_namespace["object"] = o;
py::object ignored = py::exec(
"object.f()\n",
main_namespace, main_namespace
);


Currently this yields:
TypeError: No to_python (by-value) converter found for C++ type: class boost::shared_ptr<class Object>
Obviously, if I use ObjectWrap instead of Object, I don't have that problem...

Share this post


Link to post
Share on other sites
You can register both Object and ObjectWrap. You'll want to make Object noncopyable and no_init to keep Python from using it as a base class or try to create instances on its own. Giving it a funny name and an appropriate doc string would probably help too. Ex:

#include <iostream>
#include <string>

#include <boost/python.hpp>
#include <boost/smart_ptr.hpp>

namespace py = boost::python;

class Object
{
public:
Object(int _id): id(_id) {}
virtual ~Object() {}
int getId() const { return id; }

virtual int f()
{
std::cout << "Object" << std::endl;
return 0;
}
private:
int id;
};

struct Derived : Object {
Derived(int i) : Object(i) {}

virtual int f(void) { std::cout << "Derived\n"; return 0; }
};

struct ObjectWrap: public Object, public py::wrapper<Object>
{
ObjectWrap(int i) : Object(i) {}

int f()
{
if (py::override f = this->get_override("f")) return f();
return Object::f();
}

int default_f() { return this->Object::f(); }
};

BOOST_PYTHON_MODULE(Core)
{
py::class_<Object,
boost::shared_ptr<Object>,
boost::noncopyable>("__Object",
"I'm an implementation detail. Pretend I Don't exist",
py::no_init)
.add_property("id", &Object::getId)
.def("f", &Object::f)
;

py::class_<ObjectWrap,
boost::shared_ptr<ObjectWrap>,
boost::noncopyable >("Object", py::init<int>())
.add_property("id", &Object::getId)
.def("f", &Object::f, &ObjectWrap::default_f)
;
}

int main(int argc, const char* argv[])
{
Py_Initialize();

try
{
initCore();

py::object main_module = py::import("__main__");
py::object main_namespace = main_module.attr("__dict__");
py::object ignored = py::exec(
"import Core\n"
"class CustomObject(Core.Object):\n"
" def __init__(self,id):\n"
" Core.Object.__init__(self,id)\n"
" def f(self):\n"
" print \"CustomObject\"\n"
" return 0\n"
"object = CustomObject(1337)\n",
main_namespace, main_namespace
);

py::object object = main_namespace["object"];
object.attr("f")(); // -> CustomObject

boost::shared_ptr<Object> o = py::extract< boost::shared_ptr<Object> >(object);
o->f(); // -> Object (should be CustomObject?)

boost::shared_ptr<Object> o2(new Derived(1337));
main_namespace["object"] = o2;
py::object ignored2 = py::exec(
"object.f()\n",
main_namespace, main_namespace
);
}
catch(const py::error_already_set&)
{
PyErr_Print();
}

Py_Finalize();

return 0;
}

Share this post


Link to post
Share on other sites
Alright, I was thinking about doing something like that, but it seemed like a rather hackish solution. The other option was to define a conversion that "copy constructs" an ObjectWrap from an Object.

Well, in any case, here's the final product (with some pickling thrown in as well):


#include <fstream>
#include <iostream>
#include <string>

#include <boost/python.hpp>
#include <boost/smart_ptr.hpp>

namespace py = boost::python;

class Object
{
public:
Object(int _id): id(_id) {}
virtual ~Object() {}

int getId() const { return id; }

virtual int f()
{
std::cout << "Object" << std::endl;
return 0;
}
private:
int id;
};

struct ObjectWrap: public Object, public py::wrapper<Object>
{
ObjectWrap(int _id): Object(_id) {}

int f()
{
if (py::override f = this->get_override("f")) return f();
return Object::f();
}

int default_f() { return this->Object::f(); }
};

struct PickleSuite: public py::pickle_suite
{
static py::tuple getinitargs(const Object& o)
{
return py::make_tuple(o.getId());
}

static py::tuple getstate(py::object obj)
{
const Object& o = py::extract<const Object&>(obj)();

return py::make_tuple(obj.attr("__dict__"), o.getId());
}

static void setstate(py::object obj, py::tuple state)
{
Object& o = py::extract<Object&>(obj)();

py::dict d = py::extract<py::dict>(obj.attr("__dict__"))();
d.update(state[0]);
}

static bool getstate_manages_dict() { return true; }
};

BOOST_PYTHON_MODULE(Core)
{
py::class_<Object, boost::shared_ptr<Object>, boost::noncopyable>("__Object", py::no_init)
.add_property("id", &Object::getId)
.def("f", &Object::f)
;
py::class_<ObjectWrap, boost::shared_ptr<ObjectWrap>, boost::noncopyable>("Object", py::init<int>())
.add_property("id", &Object::getId)
.def("f", &Object::f, &ObjectWrap::default_f)
.def_pickle(PickleSuite())
;
}

template<class C>
void pickle(boost::shared_ptr<C> object, const std::string& file)
{
py::object pickle = py::import("cPickle");
py::object dumps = pickle.attr("dumps")(object,0);

std::ofstream os(file.c_str());
std::string buffer = py::extract<std::string>(dumps);
os << buffer;
}

template<class C>
boost::shared_ptr<C> unpickle(const std::string& file)
{
std::ifstream is(file.c_str());
std::string buffer((std::istreambuf_iterator<char>(is)), std::istreambuf_iterator<char>());

py::object pickle = py::import("cPickle");
py::object loads = pickle.attr("loads")(buffer);

return py::extract< boost::shared_ptr<C> >(loads);
}

int main(int argc, const char* argv[])
{
Py_Initialize();

try
{
initCore();
/*
// Instance from C++
boost::shared_ptr<Object> obj(new Object(1337));
*/

// Instance from Python (Derived!)
py::object main_module = py::import("__main__");
py::object main_namespace = main_module.attr("__dict__");
py::object ignored = py::exec(
"import Core\n"
"class CustomObject(Core.Object):\n"
" def __init__(self,id):\n"
" Core.Object.__init__(self,id)\n"
" def f(self):\n"
" print \"CustomObject\"\n"
" return 0\n"
"object = CustomObject(1337)\n",
main_namespace, main_namespace
);
boost::shared_ptr<Object> obj = py::extract< boost::shared_ptr<Object> >(main_namespace["object"]);

pickle<Object>(obj, "object.txt");
obj = unpickle<Object>("object.txt");
obj->f();
}
catch(const py::error_already_set&)
{
PyErr_Print();
}

Py_Finalize();

return 0;
}


Thanks again for your help!

Share this post


Link to post
Share on other sites
Ok, let's give this another shot. I'm trying to pickle an std::vector of boost::shared_ptr<Object>. By now, I've found that pickling of stl containers is not supported by boost python and needs to be implemented by hand. One approach is that taken by cctbx (scitbx). They provide conversions from python sequences to stl containers and register them with boost python. That requires a quite some code and getting down&dirty with the Python API. However, once that's done, implementing the pickle suite is very easy.

template <class C>
struct PickleSuite: public py::pickle_suite { BOOST_STATIC_ASSERT(sizeof(C)==0); };

template <typename T>
struct PickleSuite< std::vector<T> >: public py::pickle_suite
{
static py::tuple getinitargs(const std::vector<T>& o)
{
return py::make_tuple(py::tuple(o));
}
};






While experimenting with that, it came to my mind that pickling could be achieved in a much simpler way. I implemented the getstate/setstate of the pickle suite as follows:

template <class C>
struct PickleSuite: public py::pickle_suite { BOOST_STATIC_ASSERT(sizeof(C)==0); };

template <typename T>
struct PickleSuite< std::vector<T> >: public py::pickle_suite
{
static py::tuple getinitargs(const std::vector<T>& o)
{
return py::make_tuple();
}

static py::tuple getstate(py::object obj)
{
const std::vector<T>& o = py::extract<const std::vector<T>&>(obj)();

return py::make_tuple(py::list(o));
}

static void setstate(py::object obj, py::tuple state)
{
std::vector<T>& o = py::extract<std::vector<T>&>(obj)();

py::stl_input_iterator<std::vector<T>::value_type> begin(state[0]), end;
o.insert(o.begin(),begin,end);
}
};






I've tested both solutions and both of them work for simple cases like std::vector<int>. However, I still have problems with my std::vector< boost::shared_ptr<Object> >.

		// Works:
boost::shared_ptr< std::vector<int> > objects(new std::vector<int>());
objects->push_back(1);
objects->push_back(2);
objects->push_back(3);
pickle< std::vector<int> >(objects, "objects.txt");
objects = unpickle< std::vector<int> >("objects.txt");

// Fails:
boost::shared_ptr< std::vector< boost::shared_ptr<Object> > > objects(new std::vector< boost::shared_ptr<Object> >());
objects->push_back(boost::shared_ptr<Object>(new Object(1337)));
objects->push_back(boost::shared_ptr<Object>(new Object(1338)));
pickle< std::vector< boost::shared_ptr<Object> > >(objects, "objects.txt");
objects = unpickle< std::vector< boost::shared_ptr<Object> > >("objects.txt");



The result is this curious error:
Boost.Python.ArgumentError: Python argument types in
Object.__getinitargs__(Object)
did not match C++ signature:
__getinitargs__(class Object)

In order to exclude potential error sources I have removed the virtual functions/ObjectWrap and I am left with this simple class and pickle suite.

class Object
{
public:
Object(int _id): id(_id) {}
virtual ~Object() {}

int getId() const { return id; }

private:
int id;
};

template <class C>
struct PickleSuite: public py::pickle_suite { BOOST_STATIC_ASSERT(sizeof(C)==0); };

template <>
struct PickleSuite<Object>: public py::pickle_suite
{
static py::tuple getinitargs(const Object& o)
{
return py::make_tuple(o.getId());
}

static py::tuple getstate(py::object obj)
{
const Object& o = py::extract<const Object&>(obj)();

return py::make_tuple(obj.attr("__dict__"));
}

static void setstate(py::object obj, py::tuple state)
{
Object& o = py::extract<Object&>(obj)();

py::dict d = py::extract<py::dict>(obj.attr("__dict__"))();
d.update(state[0]);
}

static bool getstate_manages_dict() { return true; }
};






And here is my boost python module:

BOOST_PYTHON_MODULE(Core)
{
py::class_<Object, boost::shared_ptr<Object>, boost::noncopyable>("Object", py::init<int>())
.add_property("id", &Object::getId)
.def_pickle(PickleSuite<Object>())
;

py::class_< std::vector< boost::shared_ptr<Object> >, boost::shared_ptr< std::vector< boost::shared_ptr<Object> > > > ("std_vector_Object", py::init<>())
.def(py::init<const std::vector< boost::shared_ptr<Object> >&>())
.def(py::vector_indexing_suite< std::vector< boost::shared_ptr<Object> > >())
.def_pickle(PickleSuite< std::vector< boost::shared_ptr<Object> > >())
;
}

Share this post


Link to post
Share on other sites
It's working now. I had to set the NoProxy template parameter of the indexing suite to true.

BOOST_PYTHON_MODULE(Core)
{
py::class_<Object, boost::shared_ptr<Object>, boost::noncopyable>("Object", py::init<int>())
.add_property("id", &Object::getId)
.def_pickle(PickleSuite<Object>())
;

py::class_< std::vector< boost::shared_ptr<Object> >, boost::shared_ptr< std::vector< boost::shared_ptr<Object> > > > ("std_vector_Object", py::init<>())
.def(py::init<const std::vector< boost::shared_ptr<Object> >&>())
.def(py::vector_indexing_suite< std::vector< boost::shared_ptr<Object> >, true>())
.def_pickle(PickleSuite< std::vector< boost::shared_ptr<Object> > >())
;
}

Sometimes a solution can be so simple, yet so hard to find... :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this