If you need to export a C interface, export a C interface. Don't try to second-guess the internals of the C++ runtime. You will only know grief.
A C interface consists of pointers and functions. If you need to, say, traverse the vector contained in your object, you need a function that takes a pointer to your object and returns the size of the vector, and another that takes a pointer to your object and an index and returns the value of the vector at that index.
If you have a void* anywhere, you're doing it wrong.
1) You can not reliably send or receive stuff to/from plugins using stl containers - even if you stick to a single compiler. If you so much as mismatch a preprocessor flag of some kind that affects the compiler's stl implementation (such as _HAS_ITERATOR_DEBUGGING), it may even appear to work at first, until one day under certain circumstances it just crashes for no explainable reason. Seriously, stay away from this.
2) Unless you like the idea of locking yourself and others into a single specific compiler for the application and all plugins, don't use a c++ interface. Use a C interface. Instead of sending a vector, send a pointer and a count of elements. Stuff like that. Even when sending POD structs without extra funniness, you will need to ensure that packing is identical between plugins and your application.
2.5) If you were just making a dll library, c++ would be slightly more viable, though you would still be restricted to certain compilers. For instance you would have to always use Visual Studio 2005 and up - stuff like that. Since this is a plugin system and you are passing things back and forth (as opposed to a library that contains everything it needs on its own), you should really, really stick with a simple C interface. Otherwise it will be very painful for reasons mentioned above. C is just not meant to interoperate with C++ (though C++ is meant to interoperate with C).
3) It may take extra work to redesign some of this, but it is not as bad as it seems. The only time you need to set up a structure to send to your plugin is immediately before making the external function call. There is no reason this should affect performance.
4) By all means feel free to experiment, but you will eventually come to these same conclusions.