Jump to content
  • Advertisement
Sign in to follow this  
Scouting Ninja

C# How to reduce data sizes?

Recommended Posts

I have been wondering about this for a while now.

Lets say I had a matrix that held location, rotation and scale, however all I would need was location(X,Y,Z), 8 directions(Every 45 angle) and 1 constant scale(Always 1).

What would be the most optimal way to store the value? So that it would use the least amount of memory and be very small so that you could send millions of them over a data packet?

Share this post


Link to post
Share on other sites
Advertisement

You can't send millions of anything in one packet.

Nor does the way you store a value necessarily relate to how you transmit a value.

One of 8 directions can be stored and transmitted in 3 bits. Scale is always 1 and therefore requires no storage or transmitted data. Location requires 3 numbers but the size of those depends on the precision of the values.

Share this post


Link to post
Share on other sites

What kind of precision do you need for your position coordinates?  What are your limits on data?  I imagine for something like this, a “snapshot frame” could send the full precision of state (X, Y, Z, rotation), while intermediary frames use less precise or fewer bits for changes between them.  If possible, I’d look into gzip compression, but I suspect it won’t have too much of an effect.

EDIT:
Some rough guesses.  Assuming some reference frame that has the full precision of the `(X, Y, Z, rotation)`, and for each frame, we use halffloat precision for deltas for a total of 27 bits/entity, then several million entities will be about 54 Mbits/update.  Or perhaps, you could use a variable number of bits and only send the components that need updating:  `(ID, (component, value), ...)`…

Share this post


Link to post
Share on other sites
8 minutes ago, Kylotan said:

You can't send millions of anything in one packet.

Interesting. Why not?

8 minutes ago, fastcall22 said:

What kind of precision do you need for your position coordinates?

Lets say the world ranges from -15 000 000 000 to +15 000 000 000 on all axis. No floating points.

12 minutes ago, Kylotan said:

One of 8 directions can be stored and transmitted in 3 bits.

What would the code for this look like?

 

Lets say the idea is the have 1 billion 2D zombies update each frame and send the update to all locations in the most optimal way. 

Share this post


Link to post
Share on other sites

The minimum amount of data you can send is 1 bit. You can't put a million bits in one packet.

You could potentially send an update that concerns millions or billions of characters and it could compress down to 1 bit, if you're lucky. That's a different problem. It's also unrealistic.

Storing 8 directions in 3 bits is just a case of reserving 3 bits of whatever storage you're using for a given number. You can flip bits in C# with the | and & operators. I suspect this is largely a waste of time compared to more productive means of reducing the cost of what you send, however.

You are not going to be able to have 1 billion zombie updates per frame without either redefining the meaning of either (a) update, (b) frame, or (c) billion.

Share this post


Link to post
Share on other sites
5 minutes ago, Kylotan said:

You are not going to be able to have 1 billion zombie updates per frame without either redefining the meaning of either (a) update, (b) frame, or (c) billion.

Would there be a reasonable amount?

I mean particles can be in the millions while holding collision data. So to what limit could we push the amount of objects if we reduce the data they use?

" 65,542 bytes in a network pack" is what I got from a quick google search. What is the rough amount of 2D zombies I could update with this?

35 minutes ago, fastcall22 said:

will be about 54 Mbits/update.

54 Mbits seem like a lot for what is needed.

 

 

Sorry if I am asking a lot of questions. I just cant understand why there are games that only update 16 characters and still send MB worth of data, while there are games updating 100 of characters for a few KB.

I assume it's because the later only send data it must and at very small precision. The question then is how? How do they reduce there data sizes?

19 minutes ago, Kylotan said:

Storing 8 directions in 3 bits is just a case of reserving 3 bits of whatever storage you're using for a given number.

I assume it has something to do with this. I have always resorted to storing data in a smaller type if I needed less precision.

For example: int MyNumber = 100; sbyte SmallerData = MyNumber.

But that means sbyte is the smallest I know how to make a number. (-128 to 127).

Share this post


Link to post
Share on other sites
6 minutes ago, Kylotan said:

To get transmission sizes down, you need to think in terms of information, not in terms of data.You're not trying to copy memory locations across the wire, you're trying to send whatever information is necessary

I feel like a idiot for not realizing this. I would never have to update the scale as it is already there and never changed. Now it makes sense, I only need to update the differences.

I could limit the data send by simply limiting the possible differences.

 

So if I had a 3rd person game where characters can jump I wouldn't need to send: bool Jump = 0; Every update. Only when the character jumps would I ever have to alert the other PC.

19 minutes ago, Kylotan said:

If you're not comfortable with bitpacking, maybe read this: https://gafferongames.com/post/reading_and_writing_packets/

I don't think I fully understand this yet. But a quick read of the link shows that it is more or less what I was thinking about. Thanks for all of the help.

Share this post


Link to post
Share on other sites

Furthering what Kylotan said about only sending the information needed rather than all the data:

If you for some reason want a million zombies on a bunch of clients and a server, I'm guessing the zombies are AI. In which case, if you calculate the exact same numerical zombie AI simulation etc on each client, in theory you only need to send the things that might affect the zombie simulation (i.e. the real players and their inputs). This is a similarish idea to client side prediction, and afaik is useful for RTS games. Presumably you have to be very careful about your simulation to make sure it runs identical on all machines.

If you really do have lots (hoping not millions) of players, then usually each client doesn't need to know about them all, only the ones within visible range. An authoritative server might keep track of all the players, but it only needs to send a subset of these to each client. And of course in the other direction the input only comes from one player per client.

Share this post


Link to post
Share on other sites

Some crash course about bit packing (i'm using hex numbers which makes it easy to understand):

int32 a = 0x01, b = 0x02, c = 0x55;

All numbers must be >=0 and <= 0xFF (255) - for that range we need 8 bits because 2^8 = 256 possibilities.

 

int32 compressed = a | (b<<8) | (c<<16);

We get 0x00550201 by bit shifting and OR to pack 3 numbers into one. (upper most 8 bits still unused - you could pack a 4th number)

 

Now unpack:

int32 A = compressed & 0xFF; // mask out the other numbers

int32 B = (compressed>>8) & 0xFF; // bit shift and mask 

int32 C = (compressed>>16); // no mask necessary if there is no 4th number

 

What to do if our original numbers are larger, but we want to use only 8 bits?

int32 a = 0x00120004;

We could ignore less important bits:

int32 qA = a>>16; // 0x00000012

then after packing and unpacking you lost 16 bits, resulting in 0x00120000 but sometimes that's good enough. Edit: E.g. if you bin characters to smaller tiles of space, less bits become enough for the position relative to the tile, which is a form of delta compression.

 

2 hours ago, Scouting Ninja said:
2 hours ago, Kylotan said:

One of 8 directions can be stored and transmitted in 3 bits.

What would the code for this look like?

You would need to tell how you store your original data. In the worst case it's a switch with 8 cases.

 

Edited by JoeJ

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Advertisement
  • Advertisement
  • Popular Tags

  • Similar Content

    • By Woody Stevens
      Hi,
       
      I am looking for a TCP or HTTP networking library similar to Lidgren (UDP), to plug into my game that is using the Monogame framework.
       
      This is primarily for sending game map data and potentially other large messages from Server to Client.
       
      I do want to keep Lidgren for my chat messages, player position, small fast updates etc. I especially love the flow of data and the library usage in general, so any libraries of a similar style would be excellent. Preferably something open source, free and reliable.
      I also must be able to swap between localhost and an ip address with ease, like Lidgren, as I run a server for singleplayer/mp/lan.
       
      My game maps are similar to minecraft, but it is 2d and only one Z-level, so i'm sending a jagged array of Tile object data (currently only enum TileID.Grass) down the pipe to the Client. Problem is if i'm sending a large map 1024 x 1024 tiles down the to client that's quite a lot of data, and Lidgren is relatively slow to build the writes (before the message is even sent!). It is fine when i'm using smaller maps < 512 x 512 ( xTiles * yTiles ).

      I know about chunking and will look into implementing this later, whilst taking into account the user's position in the world to only send nearby chunks.
       
      An example of my code that can be slow:
      private void WriteWorld(NetOutgoingMessage outgoing) { try { var world = WorldManager.Instance.CurrentWorld; outgoing.Write(world.XTiles); outgoing.Write(world.YTiles); for (int x = 0; x < world.XTiles; x++) { for (int y = 0; y < world.YTiles; y++) { // Write Tile obj data outgoing.Write((int)world.Tiles[x][y]); // <-------- Slow here when xTiles and yTiles are each > 512 ! } } } catch (Exception ex) { // log send error } }  
      I'd love to hear from you guys, especially if any of you have come across a similar challenge.
    • By ethancodes
      I'm working on a system for my game that will allow the player to stack pick ups in a queue. As one pick up expires, the next automatically activates. I'm having an issue though where if I pick up the first one, it activates fine, but if i pick up a second directly after it, it overrides the first one, activates the second one, and then once it has run it's course, everything goes back to normal gameplay, no first pick up. I'm not sure why this is happening. Hopefully someone can spot what I'm doing wrong in my code.
      Here is the code for the pick up manager:
      // Update is called once per frame void Update () { if (pickUpQueue.Count != 0 && !pickUpActive) { pickUpActive = true; pickUpQueue[0].ActivatePickUp(); } DeactivatePickUp(); } void DeactivatePickUp () { if (pickUpQueue.Count != 0 && pickUpActive) { Destroy (pickUpQueue [0]); pickUpQueue.RemoveAt (0); pickUpActive = false; } } And here is the PickUp:
      public override void ActivatePickUp () { ball.GetComponent<Ball>().Speed = 2.0f; //increase ball speed... ball.GetComponent<Ball>().StartCoroutine(timer); //...set time that power up is active }  
      There is also a Base Pick Up:
      public void OnCollisionEnter2D (Collision2D collision) { Vector2 tweak = new Vector2 (Random.Range(0f, 0.2f),Random.Range(0f, 0.2f)); this.gameObject.GetComponent<Rigidbody2D>().velocity += tweak; //if the pickup makes contact with the paddle or ball.... if (collision.gameObject.tag == "Paddle" || collision.gameObject.tag == "Ball") { GameObject.FindObjectOfType<GameManager>().GetComponent<PickUpManager>().pickUpQueue.Add(this); Destroy(gameObject); //...and finally destroy power up object } } As a side note, I am trying to find a solution to this that will work for all of my pickups. Some pickups are ammo based, some are timed. 
    • By Hellados
      Hello guys, my name is Giorgi and i'm newbie game developer i'm learning Pixel art and after pixel art  i want learn C# and don't know how and where start i'm bad with programming language and know only HTML/CSS
    • By D34DPOOL
      Edit Your Profile D34DPOOL 0 Threads 0 Updates 0 Messages Network Mod DB GameFront Sign Out Add jobEdit jobDeleteC# Programmer for a Unity FPS at Anywhere   Programmers located Anywhere.
      Posted by D34DPOOL on May 20th, 2018
      Hello, my name is Mason, and I've been working on a Quake style arena shooter about destroying boxes on and off for about a year now. I have a proof of concept with all of the basic features, but as an artist with little programming skill I've reached the end of my abilities as a programmer haha. I need someone to help fix bugs, optomize code, and to implent new features into the game. As a programmer you will have creative freedom to suggest new features and modes to add into the game if you choose to, I'm usually very open to suggestions :).
      What is required:
      Skill using C#
      Experience with Unity
      Experience using UNET (since it is a multiplayer game), or the effort and ability to learn it
      Compensation:
      Since the game currently has no funding, we can split whatever revenue the game makes in the future. However if you would perfer I can create 2D and/or 3D assets for whatever you need in return for your time and work.
      It's a very open and chill enviornment, where you'll have relative creative freedom. I hope you are interested in joining the team, and have a good day!
       
      To apply email me at mangemason@yahoo.com
    • By Andrew Parkes
      I am a talented 2D/3D artist with 3 years animation working experience and a Degree in Illustration and Animation. I have won a world-wide art competition hosted by SFX magazine and am looking to develop a survival game. I have some knowledge of C sharp and have notes for a survival based game with flexible storyline and PVP. Looking for developers to team up with. I can create models, animations and artwork and I have beginner knowledge of C sharp with Unity. The idea is Inventory menu based gameplay and is inspired by games like DAYZ.
      Here is some early sci-fi concept art to give you an idea of the work level. Hope to work with like minded people and create something special. email me andrewparkesanim@gmail.com.
      Developers who share the same passion please contact me, or if you have a similar project and want me to join your team email me. 
      Many thanks, Andrew.

    • By mike44
      The reference assemblies for framework ".NETFramework,Version=v3.5" were not found. To resolve this, install the SDK or Targeting Pack for this framework version or retarget your application to a version of the framework for which you have the SDK or Targeting Pack installed. Note that assemblies will be resolved from the Global Assembly Cache (GAC) and will be used in place of reference assemblies. Therefore your assembly may not be correctly targeted for the framework you intend.

      Hi
      what to do with the above error in ms code/Unity3d Project on Ubuntu 18.04? I've installed it like:
      https://www.microsoft.com/net/learn/...ux/ubuntu18-04
      Many thanks
    • By Dave Haylett
      Hi.
      I have pulled in five NuGet packages for my Visual Studio 2017 project, however when I build the project, VS spits out 10 .DLL and .XML files in the root of the binary folder, to do with the packages. Can't I shove them into a \packages folder so the user doesn't see these ugly resources next to the .exe file?
      I've Googled moving the packages but the only responses seem to be around moving the installation folder of the NuGet packages on the local machine, as opposed to where VS builds them to.
    • By MoreLion
      hey all! We are looking for members for our Unity horror game! 
      Here’s the story:
      After a deadly virus plunges the world into chaos killing 85% of the human population there are now what they call “zones” these zones are watched very closely by the surviving government, people are checked every day for the virus, even if you touch the spit or any human waste or fluids of the victim who is infected, you will die. But one day, people in the west zone start to go missing, 1 woman goes outside the walls to uncover the mystery, is there more to the virus than meets the eye?, That is where your story starts.
      This game is not a long development game, I have loads other game ideas,
      I will also allow you to have a bit of creative freedom if you wish to add or share a idea!
      And no, it’s not a zombie game lol I feel like zombie games are too generic, in this game you will encounter terrifying beasts!
      There is some concept art one of our concept artists have made
      If interested email liondude12@gmail.com
    • By Victor Rodriguez
      Hi there! Is the first time that I'm posting here so I'm sorry if I'm doing it wrong ha. 
      So here it comes, my doubt is, I'm doing a game with different levels, each of these levels in one different scene. Each scene contains to cameras that you can change pressing a button. Everything works fine. 
      The only problem is that I would like it to look a bit more professional, and I would like that if you finish the level with camera2, the next level start the same way. I've been thinking about using dontdestroyonloadon both cameras, but obviously this cameras need to be attached to the player to make the movement work, what do you recommend? Sorry If I've explained it in a messy way, and feel free to dm me for anything. Thanks in advance! 
    • By MoreLion
      The Standing

      Description:
      
      After a deadly virus plunges the world into chaos killing 85% of the human population there are now what they call “zones” these zones are watched very closely by the surviving government, people are checked every day for the virus, even if you touch the spit or any human waste or fluids of the victim who is infected, you will die. But one day, people in the west zone start to go missing, 1 woman goes outside the walls to uncover the mystery, is there more to the virus than meets the eye?, That is where your story starts.

      Includes:
      A Terrifying Adventure. A Ton Of Lore To Explore. First person Shooter view point. Kill, or be killed.  

      Team Name:
      Team Rebound

      Team Structure:
      Rio Dakota (Project Creator)
      Lead,Game Design,Writer
      Bastaird (Concept Artist)
      Kat (Main Character Actor)
      Voice Actor


      Previous Work:
      Project FreeFall - Unknown If Released

      Talent Required:
      Unity Programmer(3)
      Must know how to work with other Members. must know how to use unity engine Must know how to work with modellers etc Required:Must Know C#  
       3D Artist (3D Animators, Hardsurface modeller etc etc)(3)
      Ability to export Models Experience with Unity toolset a bonus. Expected to create additional props & hard surfaces for use in environments.
      3D Character Artist (1)
      Experience with Maya Or blender Or Cinema4D required. Experience with Unity toolset is a bonus. Required to create, rig and animate player, npcs & monsters. Contact:
      E-mail:Liondude12@gmail.com
      Discord:riobio55#1958
  • Advertisement
  • Popular Now

  • Forum Statistics

    • Total Topics
      631371
    • Total Posts
      2999606
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!