• Advertisement
Sign in to follow this  

[.net] Byte[] to Int[] / Memory allocation

This topic is 3760 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Is there no way to cast from an int[] array to a byte[] array? I've tried as, implicit and explicit casts but all fail at compile time. Also if there is a way, would casting be just a simple pointer type swap ala c++ or would it be converting each element? Which brings me to another question, What's the best method for memory allocation in c#? In C++, for something like this I'd use malloc and poke the data in directly, but in C# there's no alternative I know of. Bearing in mind I want safe code. I'm creating a byte array with the size of the number of bytes I want my memory allocation to be. Any better ways?

Share this post


Link to post
Share on other sites
Advertisement
Assuming you are using C#, the ONLY way to allocate memory safely is using the new keyword.

See the BitConverter class. You are going to need to copy each eleminate at a time, but the BitConverter class will give you a platform- and endian-agnostic conversion to bytes. You will need to perform a per element array conversion.

Share this post


Link to post
Share on other sites
Quote:

Is there no way to cast from an int[] array to a byte[] array?
I've tried as, implicit and explicit casts but all fail at compile time.


Try with System.BitConverter


int i = 56;
byte[] b = System.BitConverter(i);

Share this post


Link to post
Share on other sites
Wow, so there's no way to allocate a block of memory and access with as byte/int etc.

I.e imagine an old basic bank. Where you can poke ints/bytes/long etc into it at any position.

In C# a block of memory is either int/byte and the only way to treat it differently is to first convert it?

What I want to do is something like

public void PokeInt( int ByteIndex,int value)
{
byte[] data = new byte[1024];
byte+=byteindex; //advance the byte pointer
int[] idata = (int[]) data;
idata[0] = int;
}

Nothing like that possible in c#?

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
What you want to do is an unsafe typecast, so you MUST do it in unsafe code ;)

An int[] is as related to a byte[] as a string, a delegate or a System.Windows.Forms.Form

In safe code you can only cast to types the real object type inherits from.

Share this post


Link to post
Share on other sites
You have to use an unsafe context for this type of memory manipulation.

First off, you must set "Allow Unsafe Code Blocks" in project properties.

Next, you could do something like:


public void PokeInt(int byteIndex, int byteValue)
{
byte[] data = new byte[1024];

unsafe
{
fixed (byte* bytePtr = data)
{
byte* byteIter = bytePtr + byteIndex;

byte[] intBytes = BitConverter.GetBytes(byteValue);

*(byteIter)++ = intBytes[0];
*(byteIter)++ = intBytes[1];
*(byteIter)++ = intBytes[2];
*(byteIter)++ = intBytes[3];
}
}
}


Or something like:


public void PokeInt(int byteIndex, int byteValue)
{
byte[] data = new byte[1024];

unsafe
{
fixed (byte* bytePtr = data)
{
byte* byteIter = bytePtr + byteIndex;
*((int*)byteIter) = byteValue;
}
}
}


Just depends on the syntax you prefer.


Hope that helps you!

~Graham

Share this post


Link to post
Share on other sites
You can also do somehting like this (copy, not cast):

byte[] src = new byte[5*4];
int[] tgt = new int[5];
GCHandle h = GCHandle.Alloc(tgt,GCHandleType.Pinned);
Marshal.Copy(src,0,h.AddrOfPinnedObject(),src.Length);
h.Free();

Without going unsafe. It should work in reverse too. Be aware the that the byte order may be different than you expect. With a few changes you could eaisly throw primitives and structs into byte array.

If you want to look at the same data in two different ways you can use a C# union:

[StructLayout(LayoutKind.Explicit, Size=20)]
struct IntByteUnion {
[FieldOffset(0)]unsafe fixed int[5] intData;
[FieldOffset(0)]unsafe fixed byte[20] byteData;
}

But, you must be using dotnet 2.0(for fixed arrays), know how big your data is going to be and access you data from an unsafe context using a fixed block.

If you want to allocate some unmanaged memory take a look at stackalloc, Marshal.AllocHGlobal and Marshal.AllocCoTaskMem. The last two will both execute without an unsafe context and the memory can be manipulated via Marshal in a safe context, but I am not sure what sorts of checking Marshal does and if it really makes this sort of operation 'safe'.

Share this post


Link to post
Share on other sites
I think a MemoryStream wrapped with a BinaryWriter will achieve what you're after. You use it like this:

private MemoryStream memStream;
private BinaryWriter binWriter;

public void Setup(byte[] data)
{
memStream = new MemoryStream(data);
binWriter = new BinaryWriter(memStream);
}

public void PokeInt(int byteIndex, int intValue)
{
binWriter.Seek(byteIndex, SeekOrigin.Begin);
binWriter.Write(intValue);
}



Share this post


Link to post
Share on other sites
Thanks for the help. Just to clarify, is unsafe code in a .dll result in any apps using the dll needing to be marked as allowed to run unsafe code too?

Or the memory writer idea, will that be much slower than the unsafe direct method?

Share this post


Link to post
Share on other sites
Quote:
Original post by ScopeDynamo
Thanks for the help. Just to clarify, is unsafe code in a .dll result in any apps using the dll needing to be marked as allowed to run unsafe code too?

I *think* they do, but I'm not 100% on this.

Quote:
Original post by ScopeDynamoOr the memory writer idea, will that be much slower than the unsafe direct method?

While I imagine the unsafe code would be slightly faster, I certainly wouldn't use it unless I absolutely need the extra speed. IMHO just go with the MemoryStream for now, and if you find it is causing speed problems switch over to the unsafe method. So long as you don't allow the MemoryStream or BinaryWriter to be accessed directly (ie. expose your own interface to it and not the implementation), changing it over would be very easy.

Share this post


Link to post
Share on other sites
Thanks. I've just tried the memorystream idea and although it works fine for 'poking' the data into the byte array, it's more problamatic when reading data out from it.
Because I need to mirror the functionality both ways. PokeInt/PeekInt,PokeByte/PeekByte, but even a binary writer lacks a 1 for 1 match for the write method.

So is there anyway to do that?

binReader = new BinaryReader(memStream)

binReader.Seek( byteIndex,SeekOrigin.Begin);
return(int)binReader.Read();

Or am I back to using unsafe code if I want to do it?

Share this post


Link to post
Share on other sites
You can use an unsafe dll from a safe dll, the only issues come from code access security where you have situations that you demand 100 percent "safe" code.

And I agree, use unsafe code only when you have to. There are three situations, never ever use it any other time:

* Performance critical applications (as close to "real time" as you can get)

* When you need direct access to memory or structures on disk

* When dealing with legacy Win32 api calls (though certain calls can be handled effectively with P\Invoke)

~Graham

Share this post


Link to post
Share on other sites
Quote:
Original post by ScopeDynamo
So is there anyway to do that?

binReader = new BinaryReader(memStream)

binReader.Seek( byteIndex,SeekOrigin.Begin);
return(int)binReader.Read();

Or am I back to using unsafe code if I want to do it?

Yep, BinaryReader is the way to do it. You've *almost* got it right [smile]

binReader = new BinaryReader(memStream);

binReader.Seek(byteIndex,SeekOrigin.Begin);
return binReader.ReadInt32();

With the BinaryWriter it has many overloads of the Write() method for the various built-in types, but since you can't overload on the return type the BinaryReader has different method names (each prefixed with 'Read') for each type it can read.

Share this post


Link to post
Share on other sites
It looks like you actually CAN directly access a byte[] as int[] in a SAFE and efficient way. The only difference in the IL below is ldelem.i4 vs. ldelem.u1, so it should be pretty close to the metal. One quirk, is that the array length will always be expressed in the type used to allocate the array; as long as you allocate in a smaller type, you won't hit any IndexOutOfRange problems.

I have verified that the performance of repeatedly scanning a large byte[] as int[] compared to scanning a native int[] is identical when accessing as myArray[i]. Using an unsafe pointer was 5% faster. Incidentally, using foreach (int i in myArray) on a native int[] is actually as fast as using an unsafe pointer. Due to the length reporting problem, foreach on a byte[] cast to int[] (as below) throws an AccessViolationException.

Who knows how safe/portable this method is, but I'm planning to use it to reinterpret my custom struct array as bytes and vice versa.

[StructLayout(LayoutKind.Explicit)]
struct MyUnion2
{
[FieldOffset(0)]
public int[] m_stri;
[FieldOffset(0)]
public char[] m_strc;
[FieldOffset(0)]
public byte[] m_strb;
}

static void Main(string[] args)
{
MyUnion2 xx = new MyUnion2();

int[] myints = new int[] { (int)0x12345678};
byte[] mybytes = new byte[] { 0xfa, 0xfb, 0xfc, 0xfd, 0xaa,0xbb};

xx.m_strb = mybytes;

int z = xx.m_stri[0]; // returns 0xfafbfcfd
z = xx.m_stri[1]; // Surprisingly, no exception, returns 0x0000bbaa

byte a = xx.m_strb[0]; // returns 0xfa
a = xx.m_strb[1]; // returns 0xfb

foreach (byte x in xx.m_strb) {} // works
foreach (int x in xx.m_stri) {} // throws an access violation
}

L_0039: ldfld int32[] ConsoleApplication1.MyUnion2::m_stri
L_003e: ldc.i4.0
L_003f: ldelem.i4

L_004d: ldfld uint8[] ConsoleApplication1.MyUnion2::m_strb
L_0052: ldc.i4.0
L_0053: ldelem.u1

[Edited by - kganjam on October 1, 2007 7:05:06 PM]

Share this post


Link to post
Share on other sites
Quote:
Original post by gwihlidal
* Performance critical applications (as close to "real time" as you can get)

*Coughs*

Share this post


Link to post
Share on other sites
Drifting off-topic, but unsafe code is a valid optimization in some situations. An actual example: using an unsafe fixed construct to marshal managed arrays to unmanaged functions provides 20x faster call performance, compared to using a 'safe' GCHandle. A valid optimization - it is used in Tao's and OpenTK's OpenGL bindings.

Moreover, reinterpreting a byte array as an int (*(int*)) is probably faster than copying memory to achieve the same. The point here is that you have to measure and choose the fastest way (which as Washu's journal points out, isn't always obvious!)

Share this post


Link to post
Share on other sites
Quote:
Original post by Fiddler
Drifting off-topic, but unsafe code is a valid optimization in some situations. An actual example: using an unsafe fixed construct to marshal managed arrays to unmanaged functions provides 20x faster call performance, compared to using a 'safe' GCHandle. A valid optimization - it is used in Tao's and OpenTK's OpenGL bindings.


Then use marshalas, unless I'm missing something I got nowhere near a 20x hit.

The MemCpy function is just c's memcpy sat in a dll, and the results I got were as follows.

test1() - 25.27% - 1215.6ms
test3() - 24.54% - 1180.7ms
test4() - 24.43% - 1175.6ms
test2() - 24.29% - 1168.6ms

so the slowest was marshalled with unmanaged code security and the fastest was marshalled without unmanaged code security :), probably not the greatest of tests maybe, especially as if I rerun the test it can yield different results, 1324 for example, but the differences between them all is hardly anything.


[DllImport("testdll", EntryPoint = "MemCpy")]
public static extern unsafe void MemCpy1([MarshalAs(UnmanagedType.LPArray)] byte[] srcdata,[MarshalAs(UnmanagedType.LPArray)] byte[] dstdata, int size);

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("testdll", EntryPoint = "MemCpy")]
public static extern unsafe void MemCpy2([MarshalAs(UnmanagedType.LPArray)] byte[] srcdata, [MarshalAs(UnmanagedType.LPArray)] byte[] dstdata, int size);

[DllImport("testdll", EntryPoint = "MemCpy")]
unsafe public static extern void MemCpy3( byte *srcdata, byte *dstdata, int size);

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport("testdll", EntryPoint = "MemCpy")]
unsafe public static extern void MemCpy4( byte *srcdata, byte *dstdata, int size);


static void test1(byte[] b)
{
for (int i = 0; i < 100000; i++) MemCpy1(b, b, b.Length);
}
static void test2(byte[] b)
{
for (int i = 0; i < 100000; i++) MemCpy2(b, b, b.Length);
}
unsafe static void test3(byte[] b)
{
fixed (byte* bp = b) for (int i = 0; i < 100000; i++) MemCpy3(bp, bp, b.Length);
}
unsafe static void test4(byte[] b)
{
fixed (byte* bp = b) for (int i = 0; i < 100000; i++) MemCpy4(bp, bp, b.Length);
}



[STAThread]
unsafe static int Main(string[] cmdLine)
{
byte[] b1 = new byte[100000];
test1(b1);
test2(b1);
test3(b1);
test4(b1);
return 0;
}







So I guess unless mono doesn't support marshalas (I've never used mono) then I can understand why it uses the method.

and for the OP, I'd use unions but it seems that you may want to cross int boundries so gwihlidal's method is probably better. not sure what you're trying to do or what would make you need to do a mishmash of byte[]/int[] though.

Share this post


Link to post
Share on other sites
Quote:
and for the OP, I'd use unions but it seems that you may want to cross int boundries so gwihlidal's method is probably better. not sure what you're trying to do or what would make you need to do a mishmash of byte[]/int[] though.

Unions certainly seem like the best solution, the only problem being byte order on big-endian hardware.

Regarding your code, I was talking about using GCHandles to perform manual marshalling (your code only tests automatic marshalling):

[SuppressUnmanagedCodeSecurity]
delegate void FooSignature(IntPtr data);
FooSignature Foo = (FooSignature)Marshal.GetDelegateForFunctionPointer(GetProcAddress("Foo"), typeof(FooSignature));

void UnsafeUseFoo(byte[] data)
{
unsafe
{
fixed (byte* data_ptr = data)
{
Foo((IntPtr)data_ptr);
}
}
}

void SafeUseFoo(byte[] data)
{
GCHandle data_ptr = GCHandle.Alloc(data, GCHandleType.Pinned);
try
{
Foo(data_ptr);
}
finally
{
data_ptr.Free();
}
}




The GCHandle way is 10-20 times slower here, *and* it allocates memory.

If you wonder why I cannot use a [DllImport] and let the runtime handle pinning, it's because Foo is not statically exported (i.e. DllImport would return a DllEntryPointNotFound exception). P/Invoking through a delegate is also faster than using a DllImport in typical jitted code.

Share this post


Link to post
Share on other sites
Quote:
Original post by Fiddler
If you wonder why I cannot use a [DllImport] and let the runtime handle pinning, it's because Foo is not statically exported (i.e. DllImport would return a DllEntryPointNotFound exception). P/Invoking through a delegate is also faster than using a DllImport in typical jitted code.


Excuse me for being ignorant, but if you can get the entry point via GetProcAddress wouldn't you also beable to get it from DllImport assuming the entry name is correct ? or is is because of a crossplatform issue where you have to dynamically load the library ?

I'm also intrigued with GetDelegateForFunctionPointer as I've never seen this function before, how faster is it compared to using DllImport ? as I guess you have the overhead calling GetProcAddress etc anyway, or is it faster because this is only done once whereas it's done all the time when calling via a DllImport, I ask this because I interface with FreeType and if I can gain speed by changing over to GetDelegateForFunctionPointer then it would be worth me looking into.


EDIT: actually, by none static I assume you to mean a class function ?

[Edited by - Niksan2 on October 2, 2007 6:44:34 AM]

Share this post


Link to post
Share on other sites
I am using delegates to load OpenGL extensions, which are only advertised by the drivers through wgl-/glxGetProcAddress. That is, you can't DllImport these extensions as their entry points can only be obtained by querying the driver (the "GetProcAddress" call was only for illustrative purposes).

It should be possible to use GetProcAddress with Freetype - I don't have any hard performance data yet, but it seems that using delegates might be a few percents faster than DllImport. If you happen to run some tests please post the results!

Also, do you plan to publish the Freetype bindings or will you use them for an internal project? I'm asking because there is interest for a Tao.Freetype binding.

Share this post


Link to post
Share on other sites
Well, my interface to FreeType is just a wrapper to calls I use, which is essentially Load and Load_Char (getting the pixels) as you know the FreeType API is somewhat huge, but if I do have some freetime and nobody else does it I could do a proper interface.

I'll be doing some tests this afternoon regarding invoke performance and will post back later.

Share this post


Link to post
Share on other sites
Quote:
Original post by Washu
Quote:
Original post by gwihlidal
* Performance critical applications (as close to "real time" as you can get)

*Coughs*


Indeed, I'm trying to understand the intent behind the code the OP is trying to create. Seems to me that people are trying very hard to graft a C++ way of doing something on to C#.

Share this post


Link to post
Share on other sites
Quote:
Original post by paulecoyote
Indeed, I'm trying to understand the intent behind the code the OP is trying to create. Seems to me that people are trying very hard to graft a C++ way of doing something on to C#.


I guess it's the hurdle of coming from c++ to c#, what with syntax being similiar I guess your c++ instincts kick in when starting out with c#.

Fiddler:

Did a quick test, there doesn't seem to be anything between the two methods, using the MemCpy sample again I had a DllImport version and one using a delegate, calling it a million times in a loop yielded the following.

delegate: 50.40% - 11539ms
dllimport: 49.09% - 11519ms

These were done on a level playing field, so both had used the fixed keyword and both using unsafe context, so I guess as far as invoking goes they're pretty much the same. (bear in mind I'm no master at benchmarking)

Share this post


Link to post
Share on other sites
Can you retest, reducing the size of the array to 4 or even 1? This will pronounce the difference in call performance, if any. Testing with many items will tend to wash out the difference.

Share this post


Link to post
Share on other sites
You're correct it does :/

with array size set to 4:

DllImport: 30.43% 235.01ms
delegate : 59.89% 462.62ms

setting array size to 1 yields roughly the same results, the DllImport also uses auto marshalling and doesn't suppress code security.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement