Lua

I would say Lua JIT would be the way to go and then you can drop Mono completely.

I never worked with Lua, but worked with mono for a considerable while, and I loved most of it. I just had a base class that dealt with the object glue (store a pointer to the native object, get the native object from the mono object and that kind of stuff) and passed structs by ref between native<->script with no problem at all. Even arrays were pretty easy to pass from C++ to Mono

What kind of datatypes are you having trouble with?

You mean pointers to structs or to objects?

Here's an example of me passing a pointer to a C++ on-stack struct:

bool RayCastResults::call_internalAddResultOrdered( RayCastResult& result )
{
	MonoObject* excep = nullptr;

	MonoObject* resObject = mono_value_box( _domain, _classResult, &result );

	_method_internalAddResultOrdered( _obj, resObject, &excep );
	if( excep )
	{
		mono_print_unhandled_exception( excep );
		return false;
	}

	return true;
}

"_classResult" is: "_classResult = mono_class_from_name_case( img, "Engine.Physics", "RayCastResult" );"

Which on C# side is:

using System;
using System.Collections.Generic;

namespace Engine
{
	namespace Physics
	{
		public class RayCastResults
		{
			private void internal_AddResultOrdered( RayCastResult result )
			{
				...
			}
		}

		public struct RayCastResult
		{
			public override string ToString()
			{
				return string.Format("[RayCastResult] distance: {0}", distance);
			}

			public float distance;
			public CollisionObject body;
			public Shape shape;
			public Vector3 normalWorld;
			public Vector3 pointWorld;
		}
	}
}

If passing a pointer to an object, you just need to pass the MonoObject* pointer. Sometimes it's a good idea to have internal methods on C# side where you do some processing on the C++ passed data before calling the real final method.

I just noticed I posted an example passing the struct by value instead of reference. I don't have the code at hand but as far as I remember, to pass as reference you just pass a normal C++ pointer as an argument to the method.

What about if I keep Mono for the game mechanics, like generating the world and the game logic, and Lua for objects/actors which do not need complicated tasks.

Like I said, I never worked with Lua, so I'm not familiar with it's strengths or weaknesses. However, it looks weird to me that different parts of the engine are scripted with different languages.

I don't know if it applies to your project, but I'd abstract away all interactions with scripting so you could script any part of the engine with any or multiple implementations (including Mono, Lua and others) with relative ease. Maybe this way you could stick to just one of the scripting engines like you want and implement an other at a later stage if the need arises.

Regarding "Lua vs Mono" I believe you'll have to tell more about the kind of project you're working on.