• Advertisement
Sign in to follow this  

C# How to reduce data sizes?

Recommended Posts

I have been wondering about this for a while now.

Lets say I had a matrix that held location, rotation and scale, however all I would need was location(X,Y,Z), 8 directions(Every 45 angle) and 1 constant scale(Always 1).

What would be the most optimal way to store the value? So that it would use the least amount of memory and be very small so that you could send millions of them over a data packet?

Share this post


Link to post
Share on other sites
Advertisement

You can't send millions of anything in one packet.

Nor does the way you store a value necessarily relate to how you transmit a value.

One of 8 directions can be stored and transmitted in 3 bits. Scale is always 1 and therefore requires no storage or transmitted data. Location requires 3 numbers but the size of those depends on the precision of the values.

Share this post


Link to post
Share on other sites

What kind of precision do you need for your position coordinates?  What are your limits on data?  I imagine for something like this, a “snapshot frame” could send the full precision of state (X, Y, Z, rotation), while intermediary frames use less precise or fewer bits for changes between them.  If possible, I’d look into gzip compression, but I suspect it won’t have too much of an effect.

EDIT:
Some rough guesses.  Assuming some reference frame that has the full precision of the `(X, Y, Z, rotation)`, and for each frame, we use halffloat precision for deltas for a total of 27 bits/entity, then several million entities will be about 54 Mbits/update.  Or perhaps, you could use a variable number of bits and only send the components that need updating:  `(ID, (component, value), ...)`…

Share this post


Link to post
Share on other sites
8 minutes ago, Kylotan said:

You can't send millions of anything in one packet.

Interesting. Why not?

8 minutes ago, fastcall22 said:

What kind of precision do you need for your position coordinates?

Lets say the world ranges from -15 000 000 000 to +15 000 000 000 on all axis. No floating points.

12 minutes ago, Kylotan said:

One of 8 directions can be stored and transmitted in 3 bits.

What would the code for this look like?

 

Lets say the idea is the have 1 billion 2D zombies update each frame and send the update to all locations in the most optimal way. 

Share this post


Link to post
Share on other sites

The minimum amount of data you can send is 1 bit. You can't put a million bits in one packet.

You could potentially send an update that concerns millions or billions of characters and it could compress down to 1 bit, if you're lucky. That's a different problem. It's also unrealistic.

Storing 8 directions in 3 bits is just a case of reserving 3 bits of whatever storage you're using for a given number. You can flip bits in C# with the | and & operators. I suspect this is largely a waste of time compared to more productive means of reducing the cost of what you send, however.

You are not going to be able to have 1 billion zombie updates per frame without either redefining the meaning of either (a) update, (b) frame, or (c) billion.

Share this post


Link to post
Share on other sites
5 minutes ago, Kylotan said:

You are not going to be able to have 1 billion zombie updates per frame without either redefining the meaning of either (a) update, (b) frame, or (c) billion.

Would there be a reasonable amount?

I mean particles can be in the millions while holding collision data. So to what limit could we push the amount of objects if we reduce the data they use?

" 65,542 bytes in a network pack" is what I got from a quick google search. What is the rough amount of 2D zombies I could update with this?

35 minutes ago, fastcall22 said:

will be about 54 Mbits/update.

54 Mbits seem like a lot for what is needed.

 

 

Sorry if I am asking a lot of questions. I just cant understand why there are games that only update 16 characters and still send MB worth of data, while there are games updating 100 of characters for a few KB.

I assume it's because the later only send data it must and at very small precision. The question then is how? How do they reduce there data sizes?

19 minutes ago, Kylotan said:

Storing 8 directions in 3 bits is just a case of reserving 3 bits of whatever storage you're using for a given number.

I assume it has something to do with this. I have always resorted to storing data in a smaller type if I needed less precision.

For example: int MyNumber = 100; sbyte SmallerData = MyNumber.

But that means sbyte is the smallest I know how to make a number. (-128 to 127).

Share this post


Link to post
Share on other sites
6 minutes ago, Kylotan said:

To get transmission sizes down, you need to think in terms of information, not in terms of data.You're not trying to copy memory locations across the wire, you're trying to send whatever information is necessary

I feel like a idiot for not realizing this. I would never have to update the scale as it is already there and never changed. Now it makes sense, I only need to update the differences.

I could limit the data send by simply limiting the possible differences.

 

So if I had a 3rd person game where characters can jump I wouldn't need to send: bool Jump = 0; Every update. Only when the character jumps would I ever have to alert the other PC.

19 minutes ago, Kylotan said:

If you're not comfortable with bitpacking, maybe read this: https://gafferongames.com/post/reading_and_writing_packets/

I don't think I fully understand this yet. But a quick read of the link shows that it is more or less what I was thinking about. Thanks for all of the help.

Share this post


Link to post
Share on other sites

Furthering what Kylotan said about only sending the information needed rather than all the data:

If you for some reason want a million zombies on a bunch of clients and a server, I'm guessing the zombies are AI. In which case, if you calculate the exact same numerical zombie AI simulation etc on each client, in theory you only need to send the things that might affect the zombie simulation (i.e. the real players and their inputs). This is a similarish idea to client side prediction, and afaik is useful for RTS games. Presumably you have to be very careful about your simulation to make sure it runs identical on all machines.

If you really do have lots (hoping not millions) of players, then usually each client doesn't need to know about them all, only the ones within visible range. An authoritative server might keep track of all the players, but it only needs to send a subset of these to each client. And of course in the other direction the input only comes from one player per client.

Share this post


Link to post
Share on other sites

Some crash course about bit packing (i'm using hex numbers which makes it easy to understand):

int32 a = 0x01, b = 0x02, c = 0x55;

All numbers must be >=0 and <= 0xFF (255) - for that range we need 8 bits because 2^8 = 256 possibilities.

 

int32 compressed = a | (b<<8) | (c<<16);

We get 0x00550201 by bit shifting and OR to pack 3 numbers into one. (upper most 8 bits still unused - you could pack a 4th number)

 

Now unpack:

int32 A = compressed & 0xFF; // mask out the other numbers

int32 B = (compressed>>8) & 0xFF; // bit shift and mask 

int32 C = (compressed>>16); // no mask necessary if there is no 4th number

 

What to do if our original numbers are larger, but we want to use only 8 bits?

int32 a = 0x00120004;

We could ignore less important bits:

int32 qA = a>>16; // 0x00000012

then after packing and unpacking you lost 16 bits, resulting in 0x00120000 but sometimes that's good enough. Edit: E.g. if you bin characters to smaller tiles of space, less bits become enough for the position relative to the tile, which is a form of delta compression.

 

2 hours ago, Scouting Ninja said:
2 hours ago, Kylotan said:

One of 8 directions can be stored and transmitted in 3 bits.

What would the code for this look like?

You would need to tell how you store your original data. In the worst case it's a switch with 8 cases.

 

Edited by JoeJ

Share this post


Link to post
Share on other sites

If you want a billion zombies, then synchronising them over a network would be impractical... But you don't have to. You can synchronise their initial conditions and any player interactions, and then run identical updates on each client. 

https://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php

Share this post


Link to post
Share on other sites
1 hour ago, JoeJ said:

Now unpack:

int32 A = compressed & 0xFF; // mask out the other numbers

int32 B = (compressed>>8) & 0xFF; // bit shift and mask 

int32 C = (compressed>>16); // no mask necessary if there is no 4th number

This made things clearer. It turns out I have been using Bit packing for years now, except to me it's always been Texture packing.

So the same principles I would use to store textures can be used here, that makes sense. Considering that textures are made of integers.

 

23 minutes ago, Hodgman said:

But you don't have to. You can synchronise their initial conditions and any player interactions

This is what I am planning now. I can make some kind of walk states, predefined paths and checkpoints. Then just sent a integer with the number related to the path.

This would allow complex movements with a single integer per zombie.

 

 

I don't know why but this whole time I was over complicating things, some kind of mental block.:P Now everything looks so familiar and makes sense. Finally I can start trying this out for real.

Thanks for all of the help.

Share this post


Link to post
Share on other sites
25 minutes ago, Scouting Ninja said:

This made things clearer. It turns out I have been using Bit packing for years now, except to me it's always been Texture packing.

So the same principles I would use to store textures can be used here, that makes sense. Considering that textures are made of integers.

Ok, but let me make another unrelated example:

vec3 diff = findMe - nodeCenter;

int childIndex = (diff.x>0) | ((diff.y>0)<<1) | ((diff.z>0)<<2);

This way we get 3 bits from 3 comparisons and combine them to a number 0-7. If you order the children of octree in a certain order, this indexes the correct child node in an elegant and effective way. If positions are integers too, this works even without comparisons just by selecting proper bits from position (shift depends on tree level). 

 

Another one is if you have 4 packed values in one, you can add and subtract  to all of them with one op: 0x01020304 + 0x01010101 = 0x02030405. Limited, but there are practical use cases (e.g. to save registers on GPU).

 

What i mean is you should see bit packing more as a subset of bit math, which is a lot more than just compression (but not a total MUST to know).

Edited by JoeJ

Share this post


Link to post
Share on other sites
12 hours ago, Scouting Ninja said:

" 65,542 bytes in a network pack" is what I got from a quick google search. What is the rough amount of 2D zombies I could update with this?

Network packet size differs between networks. Your quoted number looks like the theoretical maximum size, which you only get with very reliable networks, eg loopback over localhost.

LANs are often around 1500. Real Internet, and mobile phone networks are even smaller. Look for the value of "mtu" of a network interface.

If you always want a single network packet for your data, you should look for the minimum guaranteed packet size rather than the largest possible packet size. The former will work on any network.

Share this post


Link to post
Share on other sites

We have had a huge ammount of AI in one game I worked on this year where we needed to simulate a 40*40km highway updating a huge ammount of vehicles. Updating a million AI agents per frame is not to solve without a huge server farm I think but why do you want to do this?

Lets assume you have a world where each player is able to see all zombies all the time, even then a zombie will typically not do anything every frame rather than stay arround most of the time. Zombies as every other AI needs a reaction time to seem realistic that may be a few hundrets of milliseconds so dosent even need to update every frame too. Lets assume this, then you could separate those million AI agents so that each agent has its reaction latency and update only a few of them every frame. Assuming 60 frames you have a frametime of 16.667 milliseconds per frame assuming a standard zombie reaction time of 200 milliseconds you need to update N / 11 zombies each frame. This are round about 91.000 zombies per frame.

Now I do not know anything about your gameplay but I do not think that you need to update 91.000 zombies each frame. Assuming a standard zombie size of 60*60cm would mean that you have a reagion of 2.730.000^2 cm or 27,3^2 km without any level geometry between them, a pure wall block of zombies. Even when your player has a view distance of 14 km, you wont need to simulate more than 10% of the zombies per frame in this region because any other zombie is too far away to be seen.

If I would make a game with such many AI, I would cheat wherever possible. Separate those zombies into different update loops that run every 10th frame for any AI that is visible (or faster like I did every 3rd frame for the vehicle AI depending on the driving speed), run your update every 20th frame for each AI that is near the player but not visible and also every 100th frame for the rest. I would do this as a sub task so every update has enought time to run and collect updates for your tasks like a packet of 20 or 50 agents data to continiously send over the network.

As I do not think you need to rotate in all three directions, you could pass one rotation value as 3 bit, one ID value as 20 bit and 35 bit for X/Z axis (Y axis is much less, maybe 16 bit) would mean 14 byte (112 bit)  per agent. A 512 byte package can then send updates for 35 agents including some overhead bytes.

Then you should reduce this too by only send updates for agents that have changed there state. I think in a real game this would reduce network traffic to a few kilobytes per second

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

  • Advertisement
  • Advertisement
  • Popular Tags

  • Advertisement
  • Popular Now

  • Similar Content

    • By Timothy Sharp
      just to test out my animations, let's say i had a trip animation. How would I code a script to where my character would fall it mud and their clothes would become all muddy? I saw this in a game on Unreal 3 but is this possible with Unity?
    • By Manuel Berger
      Hello fellow devs!
      Once again I started working on an 2D adventure game and right now I'm doing the character-movement/animation. I'm not a big math guy and I was happy about my solution, but soon I realized that it's flawed.
      My player has 5 walking-animations, mirrored for the left side: up, upright, right, downright, down. With the atan2 function I get the angle between player and destination. To get an index from 0 to 4, I divide PI by 5 and see how many times it goes into the player-destination angle.

      In Pseudo-Code:
      angle = atan2(destination.x - player.x, destination.y - player.y) //swapped y and x to get mirrored angle around the y axis
      index = (int) (angle / (PI / 5));
      PlayAnimation(index); //0 = up, 1 = up_right, 2 = right, 3 = down_right, 4 = down

      Besides the fact that when angle is equal to PI it produces an index of 5, this works like a charm. Or at least I thought so at first. When I tested it, I realized that the up and down animation is playing more often than the others, which is pretty logical, since they have double the angle.

      What I'm trying to achieve is something like this, but with equal angles, so that up and down has the same range as all other directions.

      I can't get my head around it. Any suggestions? Is the whole approach doomed?

      Thank you in advance for any input!
       
    • By Alexander Nazarov
      Hello. I'm newby in Unity and just start learning basics of this engine. I want to create a game like StackJump (links are below). And now I wondering what features do I have to use to create such my game. Should I use Physics engine or I can move objects changing transform manually in Update().
      If I should use Physics can you in several words direct me how can I implement and what I have to use. Just general info, no need for detailed description of developing process.
      Game in PlayMarket
      Video of the game
    • By Dave Haylett
      Hi all. My project is coming along wonderfully, and am starting to consider alpha deployment, and would like your advice.
      My project need access to 10,000 small PNG image files at runtime, each is only a few kilobytes each, which during development I used to load in directly from a fixed path on my HDD whenever one was needed (obviously not a solution for go-live), using something like this:
      img = new WriteableBitmap(new BitmapImage(new Uri(@screenshotsPath + filename)));
      The image would then be blitted onto a buffer screen, etc. etc. At a time, a few dozen would be being used.
      Now I'm thinking about deployment, and also when I produce an update to my app, there could be more images to add to the folders. So I'm considering the best way of a) deploying the images to the user as part of the project, and b) how to most easily handle updates to the app, whereby more images will be added.
      I have just experimented with adding them all as a Resource (!). This inflated the exe from 10mb to 100mb (not a major problem), increased the compile time from 3 secs to 30 secs (annoying), increased RAM usage from 500mb to 1.5gb (not a major problem either), but means that it solves my fixed directory issue, distribution issue, and update issue, simply by having the files all stuck into the executable. Here's the new code I'm using:
      img = BitmapFactory.FromResource("Shots/" + filename);
      The next thing I was going to try was to mark them as Content > Copy if Newer. This would resolve the executable size and RAM usage (and also the directory issue as well), however it seems that I'd need to highlight them all, and move them from Resource to Content. As an up-front job this isn't too bad, but as I add new images to the project, I'll need to go in and do this every time, which gets annoying, as the VS2015 default is Resource. Also, I'm not sure how this would work in terms of updates. Would something like ClickOnce deployment recognise new PNGs and install them to the users?
       
      I also have 3,000 ZIP files (~500kb each) which also need deploying and updating in the same way. These are currently read directly from my HDD until I can find a permanent solution for adding these to the project as well.
      Can anyone thing of a better way of doing what I'm trying to achieve?
      Thanks for any help folks.
       
    • By Felis Nigripes
      I'm doing a test quest.
      The player gets a quest from an NPC to bring him fish.

      Once the player picks up the fish, the original NPC gets replaced by a new one with a new conversation trigger. The NPC tells the Player "Well done" and should give 200xp.

      The script tells the xp counter to go up by making a reference to the gameobject that holds the text component
       
      But it throws this error:
       

       
      I'm aware that the error may hide in plain sight. I just have to sort this out, since I'm writing the AI at the same time, and the time it takes to resolve everyone of these errors is tremendous.
      Plus, I think I'll learn something. I've been having trouble with some basic functionalities recently. There might be something wrong with my understanding on how programming works.
       
      Glad if someone could help (:
       
       
       
      Edit: I'm fully aware that the update function requires an input. I call the function in the editor when the dialogue ends, it still doesn't work.
       
  • Advertisement