• Advertisement
Sign in to follow this  

x64 support & late binding of classes

This topic is 4544 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi, first off thanks for such a nice scripting implementation & for sticking with a known syntax. I have a few questions... First off, has any work been done on x64 support? If nothing, what (in code terms) needs to change to support it natively (primarily its calling conventions)? Also, out of interest, has anyone tried tried adapting AS for the PPC ISA? Additionally, say for instance I have a script that creates an native object dynamically but does not register the properties & methods of that object until an instance of that object is constructed. Will the script be able to successfully "late-bind" with the newly exposed properties & methods on the object? Or must all instance properties & methods be registered before they can be declared in a script? Thanks in advance, Will.

Share this post


Link to post
Share on other sites
Advertisement
There has been some interest in support for 64bit processors, but I don't have access to any such environment to do the development. If AS_MAX_PORTABILITY is used the actual changes to add the support should be quite small, though it can be difficult to find. The major changes that I'm aware of is that pointer sizes are 64bit instead of 32bit.

Is PPC ISA a 64bit processor? If not I think you should be able to run AngelScript on it by compiling the library with the AS_MAX_PORTABILITY mode.

At the moment a script is compiled, all features used has to be known. This includes object types, their behaviours, functions, etc.

It should be possible to configure the engine with null pointers, however. This will allow you to compile the scripts, save the byte code, and then load the pre-compiled byte code when you have the true engine configuration. (As far as I know, nobody has actually tested this yet, so I can't guarantee that it works without problems. Though, if there are any problems doing this, I'll make sure to fix them.)

Regards,
Andreas

Share this post


Link to post
Share on other sites
I've been taking a wonder through the source to get a feel for the changes required. As you say AS_MAX_PORTABILITY should work - bar pointer size issues. I suppose the main thing I'm interested in would be how much would need to change in order to support the standard calling conventions under x64. I've dug up the following two links from google that may prove useful:

http://msdn2.microsoft.com/library/7kcdt6fy(en-us,vs.80).aspx
http://blogs.msdn.com/freik/archive/2005/03/17/398200.aspx

Assuming as_callfunc_x86.cpp contains all of the logic it shouldn't be too hard to expose functions in a x64 binary. I'll have a go anyway...

The PPC ISA is 64bit from here on in. The big job certainly sounds like making the code 64bit safe. I'll try compiling the code to target x64 tomorrow to see what I run into.

Cheers,
Will.

Share this post


Link to post
Share on other sites
I'm pleased that you're willing to give it a try. Having the library support 64bit processors as well, would be a great contribution.

I'll take a look at the articles you mentioned as soon as I can.

If you want to support the native calling conventions, you'll need to port the functions in as_callfunc_x86.cpp. I suggest you create a new file as_callfunc_ppc64.cpp, or something like that.

You may also take a look at "Calling conventions on the x86 platform" to get an idea of things to look for when determining how calling conventions work on your compiler/platform. If you'd like to contribute information to that article, I'd be grateful as well. Maybe I'll expand it to other platforms as well.

Regards,
Andreas

Share this post


Link to post
Share on other sites
Well, partial success. Compiling the AS SDK for x64 works, the only required modifications have been defining AS_MAX_PORTABILITY & doubling the size of the char dummy[20] member of asUPtr (obviously as normal pointers have doubled in size). Although I haven't checked to see if this is larger then required, but for now it just works.

Admittedly the script I've tested it with is very simple & without any handles but I wanted to start off without too much complexity.

float global_var = 128;

void DoSomething()
{
global_var *= 2;
print(global_var); // Implemented as a generic function
}

The next step I'm going to try is to see if I can translate the as_callfunc_x86.cpp to a as_callfunc_x64.cpp. The biggest issue I've run into so far is that the x64 compiler included with VS2005 doesn't support inline assembly. So I'm going to have to do a bit of MASM work. Still, all good fun.

If I learn enough about the calling conventions used I may well put together a small document for you to put online. It can only help. Although it seems Microsoft & AMD have been fairly proactive on the subject themselves.

Will.

Share this post


Link to post
Share on other sites
Theres a rather large set of test apps included in the SDK. You should test the changes with them.

Share this post


Link to post
Share on other sites
This sounds great. [grin]

Did you really need to increase the size of the dummy? According to this article there are no larger method pointers than 20 bytes (even for 64bit systems). Maybe you'll just need to set it to 21 (so that the array never becomes 0).

There will likely be quite a few pointers where I've assumed a size of 4 bytes. The compiled bytecode will also be changed slightly to include direct pointers to for example global variables and functions, these will also have to be compatible with 64bit pointers.

Once you start converting the as_callfunc_x86.cpp, I suggest you start with the CDECL convention, as that is by far the most important one. Then you can move on to the CDECL_OBJLAST and CDECL_OBJFIRST. Once these 3 calling conventions are implemented, then all others can be emulated with very simple wrappers.

This is exciting, it will be great when I can officially say that the library supports 64bit systems. [grin]

Regards,
Andreas

Share this post


Link to post
Share on other sites
Quick update on progress. I've had to spend quite a bit of time getting to grips with the special prolog & epilog rules for x64 function calls. It's important as any form of stack trace & stack unwinding (in case of exceptions) need extra metadata from the assembly. Unfortunately there's not a whole lot of documentation on the subject, so I've had to do a fair bit of disassembly to work out what goes on in different cases.

It now looks like I've got it right. So now it's simply (!) a case of putting the first 4 arguments into registers & spilling the rest onto the stack prior to making the function call. Which hopefully shouldn't take too long to implement...

I double checked the asUPtr size issue. It won't compile with any size smaller then 24 bytes - but it does seem to run okay at that size. I'm not sure but I have a feeling it might be alignment related. Either way it seems that the article is incorrect, unless the compiler is making a big fuss over nothing.

Will.

Share this post


Link to post
Share on other sites
From what little I had time to read it seems that for the AMD 64bit systems, MSVC chooses fastcall as the standard calling convention. You may want to verify if there is a difference between fastcall and cdecl on that system. If there is then the constant asCALL_FASTCALL needs to be added.

The member pointer size may be 20 bytes, but that would make the other array have a size of 0, which is not allowed. Let's just leave the size of dummy at 24. The size of this type isn't really that important, since it is not used for storage.

Regards,
Andreas

Share this post


Link to post
Share on other sites
Yep, fastcall (or rather the x64 variant of it) is the calling convention for everything on x64 in windows. Which undoubtedly makes life somewhat simpler then for the x86.

Because fastcall is supported universally & is the only "legal" calling convention on x64 I'd suggest leaving the AS calling conventions as they are. The x64 compiler simply ignores any cdecl, stdcall & fastcall declarations anyway, so the x64 function calling logic should only need to worry about fastcall & ignore the asCALL_x definitions except for the cases that require attention.

I've completed the assembly required to call into any C-style x64 functions. Although one issue I ran into is the alignment of the stack that AS uses. The x64 stack is 16 byte aligned, which means to keep the assembly manageable, I have to parse the parameters off the AS stack & into a QWORD array which is then passed to the assembly, ready to copy into registers & spill onto the x64 stack.

Obviously the one big thing I haven't touched on yet is the issue of how to pass 64 bit object pointers around within AS. Might I suggest that a new define is added to AS, something like asPTR that is large enough to take the native size of a pointer on a given CPU architecture. This can then be used in those situations where pointers are passed around within AS. Obviously I don't know how you want to handle the impact this will have on bytecode instruction sizes & any thing else non-trivial that may be impacted in the change to supporting 64-bit pointers? I don't know enough at this stage to start hacking the source for it.

Anyways, I'll clean up what I've got & pass it on when it's in a reasonable state. Only issue is that VS2005 is still under beta so I'm not sure if it's legally possible to let its project files out onto the net... But even so, I can make the source available.

How do you want to tackle the pointer issue?

Cheers,
Will.

Share this post


Link to post
Share on other sites
Well, being that all calling conventions are the same on x64, then we should definitely keep the asCALL_CDECL, asCALL_STDCALL, etc as is, and have them map to the one internal calling convention ICC_FASTCALL.

I'll have to analyse the pointer issue before giving the final answer. One thing, the bytecode that is saved to disk should be compatible between all version of the library, but that doesn't stop us from having a different internal representation on 64bit processors.

If you feel up to it, you may change the library anyway you need to get it working on 64bit processors. Then you can pass me the code, and I'll find a way to unify the code so that it compiles on both 32bit and 64bit systems. You'll probably have to change the size of some bytecodes, to accomodate for the 64bit pointers.

I'm reluctant to make the AngelScript stack be align to 16byte words, as it would consume a lot more memory. To begin with what you have done is enough, but perhaps it would be possible to customize the stack alignment with a preprocessor flag for optimal behaviour.

Regards,
Andreas

PS. I wish I had a 64bit system to test this on. Anyone willing to donate one to me? [wink]

Share this post


Link to post
Share on other sites
Yep, the ICC_FASTCALL definition sounds good.

One thing that I thought of was to have the AS stack 8 byte aligned. This would aid in passing double values & (obviously) 64-bit pointers around without having unaligned data access costs & pointer size issues on x64. Although obviously it still wastes space on x86, these articles have some interesting bits on the subject of stack alignment:


http://www.fftw.org/fftw3_doc/Stack-alignment-on-x86.html


http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dv_vstechart/html/vcconwindowsdataalignmentonipfx86x86-64.asp


I'll start taking a look at pointer issues tonight. Hopefully it won't be too bad... And sorry Andreas, I've only just got access to an AMD64 system myself. Assembly is great though - the extra 8 registers make all the difference.

Cheers,
Will.

Share this post


Link to post
Share on other sites
My mistake, when you said to make the stack 8-byte aligned, I somehow thought that all variables had to be 8-byte aligned. [wink]

Of course, proper data alignment will be implemented. Some of this work was already done with the port to Dreamcast which requires aligned data in order to work. I will expand this to conform with the rules for 64bit processors as well. I consider this optimizations though, and currently that is quite low priority for me.

Thanks for the links, these are very good indeed, especially the MSDN one.

Together we will be able to make AngelScript work on both 32bit and 64bit systems. I greatly appreciate that you take the time to do this.

Regards,
Andreas

Share this post


Link to post
Share on other sites
I've decided that the next step (after finishing 2.4.0) will be a major revision of the VM. In this revision I will try to prepare the library as much as possible for 64bit support as well as add register based operations, which should make the VM much more efficient.

Things that will be done are to support variable sized pointers, aligned variables, including stack memory, etc. I'll also change the byte code instructions to read constants from a special memory buffer (that will use proper alignment) instead of having the constants inside the byte code. All byte code instructions will be 32bit in size, with room for arguments so that operations with three operands can be made.

Saving and loading byte code compiled for a 64bit machine will not be compatible on a 32bit machine, and vice versa. At least not in the beginning.

Unfortunately that ASM VM will no longer work when I make these changes. But, I'm sure that the new C++ VM will be much more efficient than the current ASM VM anyway. It will also be good to have only one VM to maintain again [wink].

64bit processors are getting more popular by the minute, and AngelScript must not be too late in supporting them or it will hurt the libraries popularity. I will do what I can to add this support, but I'll have to rely on you guys to test it for me since I don't have a 64bit processor myself.

Just thought I'd let you know what my plans are.

Regards,
Andreas

Share this post


Link to post
Share on other sites
Andreas, That's good news. I have made initial attempts to adapt the bytecode to support 64bit pointers but ran into various problems, usually due to assumptions made by other bits of code. The function calling code for x64 is still valid however - do you want a copy?

Anyway, I look forward to the new register based VM & would be more then happy to help you test & bugfix it for x64.

Cheers,
Will.

Share this post


Link to post
Share on other sites
Yes, I can imagine the problems you had.

Please send me the code for calling functions on 64bit processors. I'll include it when starting the change of the VM. You'll of course be credited for the constribution. :)

I'm sure that we will have a lot of testing to do once the work starts, but I think it will be interesting to learn more on 64bit support.



Share this post


Link to post
Share on other sites
No, JIT compilation is still far away. But I'm sure the planned changes for the VM will make future support for JIT compilation a lot easier.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement