I'm trying to make my AOT compiler "inline" native function calls so it generates a C signature based on the information available and thus bypassing the register copying mechanisms involved when making a non-generic native call. I've run into a problem I've yet to be able to figure out though.

It seems there is a difference in how AngelScript stores the parameter for a call to:
static void CopyConstructString(const string &other, string *thisPointer);
and a call to
static void func26(const MyClass3& a0);

The AOT generated code for these calls are:

// (CopyConstructString)
// _beh_0_, 10, 0, 2, 0, 0, 0
// ret: 0, 0, 0, 1, 0, 0, 0
// arg0: 67108876, 1, 7938, 0, 0, 0, 1, 0, 1, 1, 2, 2
typedef void (*funcptr)(asQWORD&, void*);
funcptr a = (funcptr)sysFunc->func;
a(**(asQWORD**)(&args[0]), obj);

// func26, 2, 0, 2, 0, 0, 0
// ret: 0, 0, 0, 1, 0, 0, 0
// arg0: 67108879, 1, 7938, 0, 0, 0, 1, 0, 1, 1, 2, 2
typedef void (*funcptr)(asQWORD&);
funcptr a = (funcptr)sysFunc->func;
a(**(asQWORD**)(&args[0]));


with the comments being
snprintf(buf, 128, "// %s, %d, %d, %d, %d, %d, %d\n", descr->GetName(), sysFunc->callConv, sysFunc->takesObjByVal, sysFunc->paramSize, sysFunc->hostReturnInMemory, sysFunc->hasAutoHandles, sysFunc->scriptReturnSize);
snprintf(buf, 128, "// ret: %d, %d, %d, %d, %d, %d, %d\n", sysFunc->hostReturnFloat, retType.IsObject(), retType.IsReference(), retType.IsPrimitive(), retType.GetSizeInMemoryDWords(), retType.GetSizeOnStackDWords(), sysFunc->hostReturnSize);
snprintf(buf, 128, "// arg%d: %d, %d, %d, %d, %d, %d, %d, %d, %d, %d, %d, %d\n", n, descr->GetParamTypeId(n), dt.IsObject(), dt.GetObjectType()->GetFlags(),dt.IsObjectHandle(), dt.IsScriptObject(), dt.IsHandleToConst(), dt.IsReference(), dt.IsPrimitive(), dt.CanBeCopied(), dt.CanBeInstanciated(), dt.GetSizeInMemoryDWords(), dt.GetSizeOnStackDWords());


So to me it looks like the string and the MyClass3 are as close to each other as can be, but the problem is that the call to CopyConstructString only works with
a(*(asQWORD*)(&args[0]), obj); // single dereference
and the call to func26 only works with
a(**(asQWORD**)(&args[0])); // double dereference

Now I hear you saying "maybe it's because you use a asQWORD rather than the real type". Unfortunately even if I put the real type in there the situation is the same:
typedef void (*funcptr)(const std::string&, std::string*);
funcptr a = (funcptr)sysFunc->func;
a(*(std::string*)(&args[0]), (std::string*) obj); // Only single dereference works
typedef void (*funcptr)(const MyClass3&);
funcptr a = (funcptr)sysFunc->func;
a(**(MyClass3**)(&args[0])); // Only double dereference works

So now the signature is a perfect match between the function pointer and the real function, agree? If you do, then to me it looks like AngelScript is indeed storing these two parameters differently on the stack, agree? How do I detect whether I should dereference once or twice?

Worth mentioning is that if I call into CallSystemFunctionNative instead of this call inlining, everything works as expected so the stack is properly set up. I tried looking in CallSystemFunctionNative for a few platforms to see if there's any special handling, but that only seems to be the case when sysFunc->takesObjByVal, which isn't the case for these two functions, or maybe the information needed isn't available when the jit compiler is invoked to compile the function?

Any ideas or suggestions?

Cheers

Nevermind, seems the argument thing was a red herring. Why it worked when calling CallSystemFunctionNative I have no idea, but it seems the stack was wrong due to broken handling of return values in the AOT code generation.