Hi guys;

I'm porting an old VB.NET app to C#/WPF. The VB.NET app uses a communications library DLL written in C, through the use of DLLImport, and I can't seem to get the correct type marshaling to work in the rewritten C# app.

Here's the original DLL method signature, as per a header file included in the SDK (please forgive the lack of variable names - it came that way.) I added the in/out comments myself, to reflect how each variable is used:
[source lang="c"]
int SendMessage(
CUSTOM_STRUCT *, //in
int, //in
unsigned char *, //in
unsigned char *, //in
unsigned long, //in
unsigned char *, //out
unsigned long *, //out
unsigned char *, //out
unsigned long *, //out
unsigned long *, //out
unsigned long ); //in
[/source]

The unsigned char * are all byte arrays for sending data and receiving responses. The unsigned long * are typically for returning how long the previous unsigned char* should be, after recieving a response. I'm not sure any of that matters, however - I think the types are sufficient enough to derive an answer.

Here is how the method was declared in the original VB.NET application: it works, but I don't like it because it's not "correct." For instance, all the byte arrays are declared to be a single ByRef Byte, but it is supposed to represent an array of bytes, and I feel that should be reflected in the method signature.
[source lang="vbnet"]
<DllImport("MyLibrary.dll")> _
Private Function SendMessage(
ByRef myStruct As CUSTOM_STRUCT, _
ByVal requestType As Short, _
ByRef messagePath As Byte, _
ByRef request As Byte, _
ByVal request_size As Integer, _
ByRef response As Byte, _
ByRef response_size As Integer, _
ByRef extended_response As Byte, _
ByRef extended_response_size As Integer, _
ByRef resultCode As Integer, _
ByVal timeout As Integer) As Integer
[/source]

Here is how I tried to declare the DllImport in the C# re-write:
[source lang="csharp"]
[DllImport("MyLibrary.dll")]
private extern static Int32 SendMessage(
ref CUSTOM_STRUCT myStruct, //in
Int32 requestType, //in
ref byte[] messagePath, //in
ref byte[] request, //in
UInt32 request_size, //in
ref byte[] response, //out
ref UInt32 response_size, //out
ref byte[] extended_response, //out
ref UInt32 extended_response_size, //out
ref Int32 resultCode, //out
UInt32 timeout); //in
[/source]

But that doesn't appear to be working. I get all sorts of unexpected behavior; the sent messages are garbage, and I occasionally get protected memory errors. Obviously I'm not marshalling these types correctly.

So I "cheated" - I modified the declaration to use IntPtr for all the reference variables, then whenever I use it I manually marshal everything to a bunch of temp IntPtrs. This appears to work, but I don't like it because it does not accurately represent the types going in and out. Not to mention the pain in the ass of manually marshaling IntPtrs and then cleaning up afterwards.
[source lang="csharp"]
[DllImport("MyLibrary.dll")]
private extern static Int32 SendMessage(
IntPtr myStruct, //in
Int32 requestType, //in
IntPtr messagePath, //in
IntPtr request, //in
UInt32 request_size, //in
IntPtr response, //out
ref UInt32 response_size, //out
IntPtr extended_response, //out
ref UInt32 extended_response_size, //out
IntPtr resultCode, //out
UInt32 timeout); //in
[/source]

Sure, that's currently my approach; just keep using what works, and wrap it in a more logically-declared method, which handles all the copying/allocating/deallocating to IntPtrs.

That said, I would still very-much like to know what exactly is wrong with my "proposed" DllImport declaration, that causes such grave marshaling errors. We have quite a few projects which rely on importing some legacy DLLs, and this knowledge could save us grief in the future, as well.

My (somewhat) educated guess is that my arrays aren't being marshaled correctly (as you said), and I need to annotate them with something along the lines of [MarshalAs(UnmanagedType.LPArray)], or something.

However, my hope was that someone would just be able to look at both the C function and my DllImport statement, and immediately offer up a "Well, thar's yer problem, right thar!"