Damn, that was exactly the problem. Quite stupidly done on my side...
But now I have different problem.
engine.register_object_type("float3", sizeof(float3), asOBJ_VALUE | asOBJ_POD | asOBJ_APP_CLASS_CA);
engine.register_function("float3 normalize(const float3 &in)", asFUNCTIONPR(normalize, (const float3&), float3));
This gives me the following message:
Don't support returning type 'float3' by value from application in native calling convention on this platform
Maybe I need different object flags?
EDIT
OK, I found the part in the documentation where it tells me to use asOBJ_APP_CLASS_ALLFLOATS!