Sign in to follow this  
AxeGuywithanAxe

C++ Reflection/Meta System

Recommended Posts

     Hey guys, so this is my second time posting on this site so here it goes. A few weeks ago I was scrolling through a few forums and saw that some people were quite frustrated with the lack of a good reflection system for c++. I thought to myself, "Whelp that can't be that hard to mimic", and began the task of creating such a system. A couple of weeks and a few more dents in the wall later I arrived at my automated reflection system that I like to call the Archetype System. The basic details of an earlier version of it can be found here; http://www.ademolathompson.com/archetypesystem , but I have overhauled it quite a bit since then. The system is inspired by the C# and UE4 reflection systems and I am quite proud of the project that it has become. Currently it supports all of the C++11 features that are currently supported in VS2013, this is because I want to later turn into into a plugin that may be run at build time just like UE4's system. To support custom features such as user created smart pointers, I designed a simple mark-up language that will create a class that mirrors the same reflection features found in the native support for c++11 smart pointers. A mark up system similar to Epic Games' may be used to declare the classes/structs/enums, and the variables/functions within them, that will be reflected to help with any memory constraints. The reflection information collected can then be exported to Json/Ini and soon xml,                                                                                                                                                                                                                Now that I've explained some of the background of the API , I will now explain why I created this post. I am one programmer , and I believe that in my finite knowledge I may have forgotten a feature that may be paramount to someone who would want to use such a system. So here I am asking this great site, filled with minds far more experienced than mine, for insight on any feature that I may have overlooked before I think of how and when I will release the API. Thank you.

Edited by AxeGuywithanAxe

Share this post


Link to post
Share on other sites
Notes from the documentation.

In C++ , the sole difference between structs and classes , is that struct member functions and variables begin with public access, while classes start as private.


Not accurate.

An added restriction is that all FClasses must inherit from FObject, while FStructs do not have an inheritance restriction.


This will perhaps be problematic with interface-based classes using multiple inheritance of interfaces or abstract bases. Yes, there are game engines that do this... a lot.

As said previously, the Archetype Reflection System only understands a subset of the C++ language. The library supports a fair majority of the new c++11 library, including the integrated smart pointers, stl, and enum classes. It does not currently support functions that share the same name, but have different signatures.


Sounds like you built your own hacky parser. There are several exquisite ones already available, including both Clang and GCC plugin systems. Did you try those? Have you seen https://bitbucket.org/dwilliamson/clreflect ?

As of now, the Archetype Reflection Standalone currently supports serialization of reflection data to windows ini and json formats. I will soon integrate XML.


A binary format would also be nice, preferably something built into the executable for things like an editor.


And lastly... does the source really need to be kept secret at this point? Planned license? Anything other than MIT/BSD makes this utterly uninteresting, at least for me.

Share this post


Link to post
Share on other sites

Thank you for your reply. What would you say is the difference between structs and classes in c++, I know the difference in C# based on allocation. But if one were to look at them as just entities, it would seem that that was the main difference. I understand that developers normally use structs for Pod containers but I am not quite sure about the technical difference. The restriction of the FCLASS, being that it must be inherited, can be easily removed with around five minutes of code so thank you for that suggestion. I built my own parser just to refresh my compiler design and because outside of the lack of template support, my parser can parse anything thrown at it while being nicely integrated into my code base. I will also look into a binary format . I will eventually release the source code under an MIT license , the only thing that is holding me back from doing it now is the lack of documentation in it, like said previously it has undertaken a large overhaul, and the fact that I may use it when I enter my Master's course in the fall. Thank you for your help and anymore would be appreciated.

Share this post


Link to post
Share on other sites

What would you say is the difference between structs and classes in c++


There's some other visibility rules, namely with bases. Saying "not accurate" was inaccurate; I should've said "incomplete" and then actually bothered to spell out the rest of it. Sorry. smile.png

The base visibility thing is probably important for a reflection library. Example:

class base {};

class derived_class : base {}; // base is private

struct derived_struct : base {}; // base is public
I also misread your docs; I thought you only called out member variables and not member functions, but you did. If you want to be totally complete, member types, member templates, member aliases, and member enums also follow the same visibility rules. smile.png

I built my own parser just to refresh my compiler design and because outside of the lack of template support, my parser can parse anything thrown at it while being nicely integrated into my code base.


This would be a show stopper for quite a few (if not all) C++ engines, especially as most of them use STL replacements that your parser is unlikely to know about (e.g. Unreal's TArray, or Boost stuff, or Bitsquid's foundation library which doesn't even pretend to be STL compatible). Not only is support of templates important for a reusable reflection library, but the ability to ensure that user-provided templates work accurately as smart pointers and containers is also important as that will likely be their primary use (though certainly not only use, at least in some engines).

The reflection systems built into engines can get away with hard-coding which special templates they work with as the engine will only support their own limited set of type templates anyway, but a generic reusable library can't make that assumption.

And even with the STL types, do you support custom allocators? Other sequence containers like std::array or std::deque? Heterogeneous containers like std::map or std::unordered_map (with custom comparators and hash functions, plus allocators) ? For the smart pointers, do you support custom deleters? Array smart pointers? Tuples and pair? What about the upcoming Library Foundations TS type templates like std::optional? And then template math library like GLM or one of the other bazillion ones out there (in this case support for only certain specializations might be enough, of course) ?

I wouldn't delay a release on any of that (in fact, release now: "release early, release often" is a frequent battle cry of Open Source) but they're things to toss on the road map.

Share this post


Link to post
Share on other sites

Ah I understand what you meant now by more to the struct//class argument, I will reword it to make it more accurate. When I stated that my library does not support templates I didn't mean that the user could not use templates, but that reflection info would not be available for the template, i.e

template<typename T>

class Foo

{

Foo();

T value;

};

, because of the lack of knowledge pertaining to the template function/class that would be used, but I believe that I may have just thought of a work around to support templates.

When it comes to Stl replacements, I designed a simple mark up language that will construct a class to allow the use of special templates containers. For example, if we use Unreal Engines TArray, the mark up would be somewhat like DEFINE_XXX_VECTOR(TArray,....) and now TArrays would be useable in my system(the system/parser would understand that it is an "alias " type that has functionality similar to vectors). Custom allocators/deallocators/deleters and heterogenous containers are also supported through the mark up system through helper objects so I believe the only issues that I may have are std::optional , tuples and pairs. The tuples/pairs should be an easy fix with just another addition to the mark up language.

I believe microsoft stated that they would not update msvc anymore towards the C++11 standard until 2014 so I believe that the lack of support for std::optional I can push until then.  Supporting a template library would be a difficult task to attempt based on the fact that one would not know each template type that is used without parsing the full source code, but I believe that the new method I thought of may be able to cover such a circumstance. I will hopefully be able to document it and clean up the code a bit for an actual release soon as long as I kick the habit of trying to be a perfectionist. Thank you for your help and if you have anymore insights I would be greatful.

Edited by AThompson

Share this post


Link to post
Share on other sites
Yeah, ok, it wasn't clear what you meant by template support.

You can't really do a whole lot with reflection of generic templates. Template specializations/instantiations require compiler support to generate the code. The best you can do is add enough markup so that the library knows that a `vector<typename T>` describes a vector and that `vector<int>` is an instance of that template.

You can test support for std::optional much sooner by just writing one yourself or using Boost's or using one of the Library Fundamentals implemented in other vendors' C++ libraries. There is plenty of code in games that is only run on the server or only on the popular mobile platforms and only needs to compile with GCC or Clang; you don't want to lock out VC++, no, but you don't want to lock out C++14 and newer features just because VC++ is still catching up. smile.png

Otherwise, sounds like most of these issues are just documentation issues and the lack of source to peruse.

Share this post


Link to post
Share on other sites

Well the mark up language creates a class that can handle all instances of UserDefinedArray<typename T>, through DEFINE_XXX_VECTOR(UserDefinedArray..) (it is only required to be written once not per template),  and is then added through an alias system to the parser. Once it's added, the parser will know that any instance of UserDefinedArray<typename T> found during the parsing phase is a defined array and will construct the Cpp code necessary to handle reflecting the value, whether it's UserDefinedArray<int> or UserDefinedArray<UserDefinedArray<UserDefinedArray<int>>> ;) . I will definitely look into std::optional and multi compiler support , but I want to make sure I have all of the random bugs that may exist worked out while using my beloved msvc haha. I will make sure to update the documentation once the coding phase is more stable and either update my website, this post or both. Thank you for your help, it's been appreciated.

Share this post


Link to post
Share on other sites

Whelp that can't be that hard to mimic

 

 

 

This made me chuckle. smile.png

 

The reality is not that it's hard to make a reflection system in C++. It's hard to make one that works really well, really safely, really effortlessly, and satisfies all the myriad competing needs individual developers have (or don't have) for such a system. It will always be this way, at least until the language gets more native support for some of the fundamental building blocks (although maybe not even then, depending). Instead, as you'll discover here (if you haven't already), you're going to be forced to choose between several arguably unfortunate design trade-offs which infuse your system with certain limitations. These will be deal-killers for some people, and not worth worrying about for others. It's why there are tons of C++ reflection libraries floating around. 

 

But that's not really want I wanted to post about. I wanted to agree with Sean on the oddity of keeping your source closed; you'll get much more in-depth critiques and suggestions if the implementation is available so people can dig through the actual implementation details and costs/benefits of your approach.

 

Also, what's with using screenshots of code on the ArchetypeSystem site? Its ugly and hard to read.

Edited by Josh Petrie

Share this post


Link to post
Share on other sites

While I appreciate the intent behind wanting a c++ reflection system, you need to understand that your goal is at cross purposes with the design of C++, both the language and the optimizer.
 
The language is built on a model of elimination. Don't include every header, only include the minimum needed to make it work. Don't instantiate every template function, only the ones required to make it work. Don't emit code for every function, inline what you can. Don't incorporate all the functions from the libraries, have the linker strip everything that is unused. Etc.
 
I also think the web site itself does a good job of covering the common difficulties that EVERY project like this has faced:

Archetype Reflection System only understands a subset of the C++ language. ... It does not currently support functions that share the same name, but have different signatures. Templated functions, structs, and classes are not supported, and macros requiring expansion are yet to be included. Archetype Classes/Structs currently only support single inheritance.

I'm guessing a whole lot of other parts of the language are not covered as well. 

 

How does it handle compiler-generated and implicit functions? How does it handle functions that emit code but are marked as deleted?  How does it handle static names -- that is, names with no external linkage? How do you handle ODR-used elided functions? How does it handle SFINAE?

 

One of the key features in modern reflections -- something present in every language I've used reflection in -- is that you can use every aspect of the class without having access to the source. That includes functions that in c++ are often inlined or elided.  For example, if the original programmer provides 50 different functions for a class but then the application developer uses only 10 different functions, every good compiler will ensure the compiled output only has those 10 used functions, and they might even be able to eliminate some of those, so maybe only 8 or 6 of those functions have a callable function. Someone wanting to use reflection would need to have access to all 50 of those functions relying only on the final binary --- except the compiler has removed them. 

 

 

 

As has been covered many times by many different groups, the only way to completely add modern reflection to C++ is through a major compiler and linker overhaul, the results of the overhaul would be code that violates many expectations and requirements of c++.  When you figure out how to enable reflection with C++'s feature of SFINAE and how to reflect C++ template metaprogramming, the entire world would like to know your solutions.

Share this post


Link to post
Share on other sites

Thank you for your responses. Like I previously stated Josh, I am currently overhauling the system to make it more efficient and adding documentation to it. Because of this I am waiting on releasing source code until the build quality is higher and any deprecated code from previous iterations are removed. I created this post to gain ideas of features that I may have overlooked so that I could have a more complete product when released. The pictures on my site are from an older version of the system and they were not placed there to show the source code, but to show how a user would use my api, somewhat similar to https://www.unrealengine.com/blog/unreal-property-system-reflection. Rob thank you for your input also, I understand that to cover the full extent of the C++ language I would need compiler/linker support, which is why I chose to take a subset that I thought would proficient enough for general use, this subset for the most part reduces the amount of support for reflecting a class/function/struct that is templated because one can not know what the actual typename of the class/struct/function will be without knowing all of the template specializations.  As far as I know, i every c++ reflection system that exists currently has issues with pruned functions and variables . The only approach that seemed reasonable would be to limit the template reflection support to a C# level.

So for instance if this is in the source header:

FCLASS()

template<typename T>

class Foo

{

FPROPERTY()

T FooValue;

};

my api can state that the class is templated, even give you the property FooValue by name,

but it would be up to the user to supply the actual template for the property when trying to access it like:

 

Foo<int> ClassObject;

FProperty* pFooValue = ClassObject->GetProperty(“FooValue”);

int actualValue = pFooValue->GetValueAs<int>();

 

and pFooValue->SetValueAs<int>(99);


This would of course limit the support of templates( i.e meta programming and etc), but game engines that currently exist have less features than my API and make due so hopefully that will be enough. I still have to update the old site because that is a two month old post and the API has had a large overhaul, it now supports multiple inheritance and etc .

Share this post


Link to post
Share on other sites

After reading what you wrote I cant help but think you are making the classic mistake of building an overengineered overgeneralized framework, without actually having a use for it. You are forced to look around and ask for more features you should add, because you actually dont need any yourself.

You better first write the user code and only then add missing features to the API, else you would waste your time on making stuff you dont need in a way thats not easily usable by anyone.

Share this post


Link to post
Share on other sites

To echo the guy above me, there's a saying: The worst code has no users. So you yourself can't possibly write a good API unless you have users to test it and provide feedback so you can iterate on the API.

 

Also some people don't like including APIs that make exorbitant use of templates since they eat up compile times. You might consider an imlementation revolving around PODs that doesn't require templates at all.

Share this post


Link to post
Share on other sites
Um, everything above, plus your not exactly blazing a trail here, I know of ONE reflection framework for C++ that I've used, plus many more. This is just googling sloppily:

Konstatin Knihznik's reflection package
Reflex from CERN
XCppRefl
Mirror

Its cool to offer code and all, but you kind of sound like ISO/IEC is just waiting for you to slide this their way.

Randy Gaul: There's no way you can know if he's making "exorbitant" use of templates. And a reflection framework is a textbook scenario for templates. Not "exorbitant" ones, but still...

Share this post


Link to post
Share on other sites

I didn't state that I was trail blazing, in fact I stated three reflection systems on my website that were my source for inspiration on creating the system. I've also stated multiple times that I will eventually release a "user" type once I am done refactoring/documenting the code. The user portion is done feature wise I was just working on making it more efficient, This is why I created this post asking for any features others would want. I have a full set of features that I would use but I know that there may be some that others may want, i.e the exporting as a binary file. My library also does not create templated code unless the user uses a property that requires templating, i.e std::vector<int>/TArray<int> so the added executable/compile time is minimal. I would not state that my reflection system is over generalized, the only "general" feature that it has is allowing users to add their own smart pointers/container so that my system can reflect the data/functions reducing the code required to obtain the property i,e:

 

turning:

 

Foo ClassObject;

FProperty* pProperty = ClassObject.GetProperty(“foovector”);

std::vector<int> Vector = pProperty->GetPropertyAs<std::vector<int>>();

Vector.push_back(5);

 

into:

 

Foo ClassObject;

FProperty* pProperty = ClassObject.GetProperty(“foovector”);

pProperty->push_back(5);

if that increases to the “generalization” of code then I shall rethink it and remove all of my container/smart pointer specialized property support.

Share this post


Link to post
Share on other sites

I'm thinking that you got what you asked for in the request, although perhaps not as positive as you would have hoped.

 

That's always a risk when asking for feedback.

 

By asking what broad features of C++ it doesn't support the answers are going to be harsh. It looks like while that was what you asked, it wasn't quite what you meant.

 

From what I was able to see, it looks like you've spent a lot of time on it. If it works for you, then great. 

Share this post


Link to post
Share on other sites

I'm okay with feedback but I stated in the first post that I have the api working at a user level and received responses telling me to finish a user level version of the application which I stated in the topic post I already completed. I also stated that My work on it was based on previously created reflection systems . The responses I got besides  Sean was that I need to create a user level version of the api and that I wasn't at the forefront of c++ reflection systems which completely contradict what I stated. This discussion was supposed to be based on the features that some people would want in a c++ reflection system, the idea of binary exporting was brought up , just encase I overlooked a certain feature. This would be the second phase of what Josh said I should do, being ask for user feedback after creating a baseline model. Your, frob, post about elide functions was educational in the sense of issues that my system would have with compiler backend optimizations and is something I have written down for further study. It is possible that I should of narrowed down the features for those reading this post to think about, as in supported export formats/compilers, to receive an answer more focused on features than common pitfalls of reflection systems or other currently available reflection systems, so this may be my error.

Share this post


Link to post
Share on other sites
Yes, you asked about the things that people would find most useful.

For me, that is the most useful thing. I use reflection quite a lot in some projects, and for me that is the biggest thing I would need. Without the ability to open a compiled library and generate a 100% usable class every time, it is not usable. Unfortunately it is also a thing that cannot be solved with the object file compilation model.

There have been various good compromises in that process over the years. COM is a very old solution to the problem and still one of the better implementations at general purpose interoperation without sacrificing performance. They (very wisely) do not attempt to abstract out every language, instead allowing component developers to publish an interface contract.

Share this post


Link to post
Share on other sites

Well thank you for your help. It is an interesting subject to talk about. I wonder how often compilation errors occur in development. Unreal Engine's reflection system is heavily tied into their blue print scripting system and it would be quite catastrophic if the compiler optimizes in a way that crashes their reflection system. Maybe it is possible that they disable any optimizations to ensure safety of the reflection code.  I wanted to use Com for my runtime scripting system, I rather like staying with c++ instead of lua, but eventually began working on other parts of my engine. I might read up on it more later though.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this