Hey guys, so this is my second time posting on this site so here it goes. A few weeks ago I was scrolling through a few forums and saw that some people were quite frustrated with the lack of a good reflection system for c++. I thought to myself, "Whelp that can't be that hard to mimic", and began the task of creating such a system. A couple of weeks and a few more dents in the wall later I arrived at my automated reflection system that I like to call the Archetype System. The basic details of an earlier version of it can be found here; http://www.ademolathompson.com/archetypesystem , but I have overhauled it quite a bit since then. The system is inspired by the C# and UE4 reflection systems and I am quite proud of the project that it has become. Currently it supports all of the C++11 features that are currently supported in VS2013, this is because I want to later turn into into a plugin that may be run at build time just like UE4's system. To support custom features such as user created smart pointers, I designed a simple mark-up language that will create a class that mirrors the same reflection features found in the native support for c++11 smart pointers. A mark up system similar to Epic Games' may be used to declare the classes/structs/enums, and the variables/functions within them, that will be reflected to help with any memory constraints. The reflection information collected can then be exported to Json/Ini and soon xml, Now that I've explained some of the background of the API , I will now explain why I created this post. I am one programmer , and I believe that in my finite knowledge I may have forgotten a feature that may be paramount to someone who would want to use such a system. So here I am asking this great site, filled with minds far more experienced than mine, for insight on any feature that I may have overlooked before I think of how and when I will release the API. Thank you.
C++ Reflection/Meta System
In C++ , the sole difference between structs and classes , is that struct member functions and variables begin with public access, while classes start as private.
Not accurate.
An added restriction is that all FClasses must inherit from FObject, while FStructs do not have an inheritance restriction.
This will perhaps be problematic with interface-based classes using multiple inheritance of interfaces or abstract bases. Yes, there are game engines that do this... a lot.
As said previously, the Archetype Reflection System only understands a subset of the C++ language. The library supports a fair majority of the new c++11 library, including the integrated smart pointers, stl, and enum classes. It does not currently support functions that share the same name, but have different signatures.
Sounds like you built your own hacky parser. There are several exquisite ones already available, including both Clang and GCC plugin systems. Did you try those? Have you seen https://bitbucket.org/dwilliamson/clreflect ?
As of now, the Archetype Reflection Standalone currently supports serialization of reflection data to windows ini and json formats. I will soon integrate XML.
A binary format would also be nice, preferably something built into the executable for things like an editor.
And lastly... does the source really need to be kept secret at this point? Planned license? Anything other than MIT/BSD makes this utterly uninteresting, at least for me.
Thank you for your reply. What would you say is the difference between structs and classes in c++, I know the difference in C# based on allocation. But if one were to look at them as just entities, it would seem that that was the main difference. I understand that developers normally use structs for Pod containers but I am not quite sure about the technical difference. The restriction of the FCLASS, being that it must be inherited, can be easily removed with around five minutes of code so thank you for that suggestion. I built my own parser just to refresh my compiler design and because outside of the lack of template support, my parser can parse anything thrown at it while being nicely integrated into my code base. I will also look into a binary format . I will eventually release the source code under an MIT license , the only thing that is holding me back from doing it now is the lack of documentation in it, like said previously it has undertaken a large overhaul, and the fact that I may use it when I enter my Master's course in the fall. Thank you for your help and anymore would be appreciated.
What would you say is the difference between structs and classes in c++
There's some other visibility rules, namely with bases. Saying "not accurate" was inaccurate; I should've said "incomplete" and then actually bothered to spell out the rest of it. Sorry.
The base visibility thing is probably important for a reflection library. Example:
class base {};
class derived_class : base {}; // base is private
struct derived_struct : base {}; // base is publicI built my own parser just to refresh my compiler design and because outside of the lack of template support, my parser can parse anything thrown at it while being nicely integrated into my code base.
This would be a show stopper for quite a few (if not all) C++ engines, especially as most of them use STL replacements that your parser is unlikely to know about (e.g. Unreal's TArray, or Boost stuff, or Bitsquid's foundation library which doesn't even pretend to be STL compatible). Not only is support of templates important for a reusable reflection library, but the ability to ensure that user-provided templates work accurately as smart pointers and containers is also important as that will likely be their primary use (though certainly not only use, at least in some engines).
The reflection systems built into engines can get away with hard-coding which special templates they work with as the engine will only support their own limited set of type templates anyway, but a generic reusable library can't make that assumption.
And even with the STL types, do you support custom allocators? Other sequence containers like std::array or std::deque? Heterogeneous containers like std::map or std::unordered_map (with custom comparators and hash functions, plus allocators) ? For the smart pointers, do you support custom deleters? Array smart pointers? Tuples and pair? What about the upcoming Library Foundations TS type templates like std::optional? And then template math library like GLM or one of the other bazillion ones out there (in this case support for only certain specializations might be enough, of course) ?
I wouldn't delay a release on any of that (in fact, release now: "release early, release often" is a frequent battle cry of Open Source) but they're things to toss on the road map.
Ah I understand what you meant now by more to the struct//class argument, I will reword it to make it more accurate. When I stated that my library does not support templates I didn't mean that the user could not use templates, but that reflection info would not be available for the template, i.e
template<typename T>
class Foo
{
Foo();
T value;
};
, because of the lack of knowledge pertaining to the template function/class that would be used, but I believe that I may have just thought of a work around to support templates.
When it comes to Stl replacements, I designed a simple mark up language that will construct a class to allow the use of special templates containers. For example, if we use Unreal Engines TArray, the mark up would be somewhat like DEFINE_XXX_VECTOR(TArray,....) and now TArrays would be useable in my system(the system/parser would understand that it is an "alias " type that has functionality similar to vectors). Custom allocators/deallocators/deleters and heterogenous containers are also supported through the mark up system through helper objects so I believe the only issues that I may have are std::optional , tuples and pairs. The tuples/pairs should be an easy fix with just another addition to the mark up language.
I believe microsoft stated that they would not update msvc anymore towards the C++11 standard until 2014 so I believe that the lack of support for std::optional I can push until then. Supporting a template library would be a difficult task to attempt based on the fact that one would not know each template type that is used without parsing the full source code, but I believe that the new method I thought of may be able to cover such a circumstance. I will hopefully be able to document it and clean up the code a bit for an actual release soon as long as I kick the habit of trying to be a perfectionist. Thank you for your help and if you have anymore insights I would be greatful.
You can't really do a whole lot with reflection of generic templates. Template specializations/instantiations require compiler support to generate the code. The best you can do is add enough markup so that the library knows that a `vector<typename T>` describes a vector and that `vector<int>` is an instance of that template.
You can test support for std::optional much sooner by just writing one yourself or using Boost's or using one of the Library Fundamentals implemented in other vendors' C++ libraries. There is plenty of code in games that is only run on the server or only on the popular mobile platforms and only needs to compile with GCC or Clang; you don't want to lock out VC++, no, but you don't want to lock out C++14 and newer features just because VC++ is still catching up.
Otherwise, sounds like most of these issues are just documentation issues and the lack of source to peruse.
Well the mark up language creates a class that can handle all instances of UserDefinedArray<typename T>, through DEFINE_XXX_VECTOR(UserDefinedArray..) (it is only required to be written once not per template), and is then added through an alias system to the parser. Once it's added, the parser will know that any instance of UserDefinedArray<typename T> found during the parsing phase is a defined array and will construct the Cpp code necessary to handle reflecting the value, whether it's UserDefinedArray<int> or UserDefinedArray<UserDefinedArray<UserDefinedArray<int>>> ;) . I will definitely look into std::optional and multi compiler support , but I want to make sure I have all of the random bugs that may exist worked out while using my beloved msvc haha. I will make sure to update the documentation once the coding phase is more stable and either update my website, this post or both. Thank you for your help, it's been appreciated.
Whelp that can't be that hard to mimic
This made me chuckle.
The reality is not that it's hard to make a reflection system in C++. It's hard to make one that works really well, really safely, really effortlessly, and satisfies all the myriad competing needs individual developers have (or don't have) for such a system. It will always be this way, at least until the language gets more native support for some of the fundamental building blocks (although maybe not even then, depending). Instead, as you'll discover here (if you haven't already), you're going to be forced to choose between several arguably unfortunate design trade-offs which infuse your system with certain limitations. These will be deal-killers for some people, and not worth worrying about for others. It's why there are tons of C++ reflection libraries floating around.
But that's not really want I wanted to post about. I wanted to agree with Sean on the oddity of keeping your source closed; you'll get much more in-depth critiques and suggestions if the implementation is available so people can dig through the actual implementation details and costs/benefits of your approach.
Also, what's with using screenshots of code on the ArchetypeSystem site? Its ugly and hard to read.
While I appreciate the intent behind wanting a c++ reflection system, you need to understand that your goal is at cross purposes with the design of C++, both the language and the optimizer.
The language is built on a model of elimination. Don't include every header, only include the minimum needed to make it work. Don't instantiate every template function, only the ones required to make it work. Don't emit code for every function, inline what you can. Don't incorporate all the functions from the libraries, have the linker strip everything that is unused. Etc.
I also think the web site itself does a good job of covering the common difficulties that EVERY project like this has faced:
Archetype Reflection System only understands a subset of the C++ language. ... It does not currently support functions that share the same name, but have different signatures. Templated functions, structs, and classes are not supported, and macros requiring expansion are yet to be included. Archetype Classes/Structs currently only support single inheritance.
I'm guessing a whole lot of other parts of the language are not covered as well.
How does it handle compiler-generated and implicit functions? How does it handle functions that emit code but are marked as deleted? How does it handle static names -- that is, names with no external linkage? How do you handle ODR-used elided functions? How does it handle SFINAE?
One of the key features in modern reflections -- something present in every language I've used reflection in -- is that you can use every aspect of the class without having access to the source. That includes functions that in c++ are often inlined or elided. For example, if the original programmer provides 50 different functions for a class but then the application developer uses only 10 different functions, every good compiler will ensure the compiled output only has those 10 used functions, and they might even be able to eliminate some of those, so maybe only 8 or 6 of those functions have a callable function. Someone wanting to use reflection would need to have access to all 50 of those functions relying only on the final binary --- except the compiler has removed them.
As has been covered many times by many different groups, the only way to completely add modern reflection to C++ is through a major compiler and linker overhaul, the results of the overhaul would be code that violates many expectations and requirements of c++. When you figure out how to enable reflection with C++'s feature of SFINAE and how to reflect C++ template metaprogramming, the entire world would like to know your solutions.
Thank you for your responses. Like I previously stated Josh, I am currently overhauling the system to make it more efficient and adding documentation to it. Because of this I am waiting on releasing source code until the build quality is higher and any deprecated code from previous iterations are removed. I created this post to gain ideas of features that I may have overlooked so that I could have a more complete product when released. The pictures on my site are from an older version of the system and they were not placed there to show the source code, but to show how a user would use my api, somewhat similar to https://www.unrealengine.com/blog/unreal-property-system-reflection. Rob thank you for your input also, I understand that to cover the full extent of the C++ language I would need compiler/linker support, which is why I chose to take a subset that I thought would proficient enough for general use, this subset for the most part reduces the amount of support for reflecting a class/function/struct that is templated because one can not know what the actual typename of the class/struct/function will be without knowing all of the template specializations. As far as I know, i every c++ reflection system that exists currently has issues with pruned functions and variables . The only approach that seemed reasonable would be to limit the template reflection support to a C# level.
So for instance if this is in the source header:
FCLASS()
template<typename T>
class Foo
{
FPROPERTY()
T FooValue;
};
my api can state that the class is templated, even give you the property FooValue by name,
but it would be up to the user to supply the actual template for the property when trying to access it like:
Foo<int> ClassObject;
FProperty* pFooValue = ClassObject->GetProperty(“FooValue”);
int actualValue = pFooValue->GetValueAs<int>();
and pFooValue->SetValueAs<int>(99);
This would of course limit the support of templates( i.e meta programming and etc), but game engines that currently exist have less features than my API and make due so hopefully that will be enough. I still have to update the old site because that is a two month old post and the API has had a large overhaul, it now supports multiple inheritance and etc .
