We have a server that has physics processing module (rigid bodies). Need to send minimal data to clients to see the same scene.

As I understand, sending positions and orientations is not good because if body moves then client gets about 40 bytes per frame, so maybe 400 bytes per second. It's just for one object. But if there's a hundred?

So good solution as I understand is to send position, orientation, linear and angular velocities. So if body has linear movement, nothing is being sent. The problem is when to send? Maybe the solution is to emulate client on server - to store those data and update like pos = pos + vel * dt, quat = ... and check the difference between real position and orientation, and between this client emulation. So if difference is more than 0.1 then update client (send all new data). This is bad with quaternions, because they are not linear and the difference |x1 - x2| + |y1 - y2| + |z1 - z2| + |w1 - w2| will be different on different angles.

So what to do?

How critical are these objects? Do all of these hundreds of objects actually impact game play? (Can players take cover behind them, run on them, etc). If not, you can just have the client guess and throw objects around based on whatever data it last got from the server. If they are, you're going to have to come up with some kind of priority scheme. You probably won't be able to send data for hundreds of objects over the network at once. What you can do is calculate the closest objects, objects the player is looking at, etc, and give them a higher priority, send their state (position, orientation, velocity) to the clients, and have the clients run physics until they get the next state. Low priority objects can be continuously upped in priority until they are finally sent and everything in the world is updated.

I didn't mean that I have millions of objects. Just hundred. If I send 40 bytes * 100 objects * 10 times per second I get 4 kb per second, but it's too much for falling and rolling balls for example. If all of them are moving on the flat ground elements, then we can send their data every 5 seconds for example, because client just makes x = x + v * dt.

Does anyone use some motions or rotations via network?

Yes, it is very common to do "baselining" where you send a compressed snapshot of the state of the object for certain times (say, once every N seconds), and then only send "change" events, like when the server detects a collision between objects, in the meanwhile.
You also need to send player inputs in a constant stream, so that player entities keep updating the simulation.

Because of the transmission latency between clients and the server, there are two main problems you'll need so solve here:
1) players colliding with other players -- you don't have perfect knowledge, so the players will see different things; you have to reconcile this
2) multiple players wanting to collide/interact with the same object. For example, picking up a health pack from the ground -- who got there first?

In the past I've had success with slerping quaternions or using a data filter with upper and lower boundary checks with wrap around. Like this: http://www.replicanet.com/assets/docs/Main/html/classRNReplicaNet_1_1DataBlock__Predict__Float.html#da3fce1d4164fc4f075257b1406ee3eb

The quaternion approach was especially good with networked physics because rotations could use deltas.

hplus0603, you always describe small manual how to make a game (something like this). I don't have problems with synchronization of object getting. I just have physics server, so if player-A-packet-for-getting-object will be received first than player-B-bla-bla-bla then server lets player A get this object.

And yes, I know that quaternions are good for rotation, because without them we won't get right processing. But what about networking? Because physics engine has it's own updating pipeline, own stepsize (ofcourse it's fixed and about 0.2 seconds). But client has about 50 frames per second, so something like q_new = 1/2 * q_old * w is not equal for different step sizes. And if we process another formulae with quaternion exponents (that's equal for different step sizes), we get not-synchronized physics on client and on server.

I thought about getting the difference of quaternions (like q_diff = q_new / q_old) on every server step and then just interpolate on client, but rotations bigger than 180 degrees per server step are not right. Maybe it's better than not-synchronized physics on client and server that makes server send more packets on constant rotation...

I don't think that will work as described. You need more smarts to make a distributed rigid body simulation works alright (this is why most multiplayer games have very little rigid body physics!)

First, the server needs to have a short time step (ideally, the same as clients), to avoid tunneling, and make simulation consistent.

Second, you want to assume that all players will simulate the same objects the same way most of the time -- the only case that can change that is player action. Thus, you want to keep track of which objects are being affected by players at what time steps, and send information about those objects.

Third, you want to snap the physics objects to the locations you get from the server, as soon as you get them (and then simulate them forward again to whatever time step you actually display to the client). The visual snap should be hidden in rendering, not in simulation; for example by calculating a delta error and lerping that error (both for angular and linear position) to 0 over a few frames of rendering.

Finally, you want to cycle through all your simulated bodies, and send a checkpoint for each body once in a while to all players, so that a body that goes out of sync will be put into sync again.

First, client has about 100 "interesting" objects. Server has millions of objects, some of them are inactive. Client just renders good quality frames (shaders, etc) and server manages all the objects, so as I described I just need to optimize sending of physics data, not managing interesting objects, not managing players synchronization in their interaction. So server steps are too big, and client makes about 50 frames per second, on some machines I get 200 frames. Server cannot simulate physics on that speed...

Problem is in several words. Server sends physics data, not much, not too often, nothing else. I talk just about minimizing of physics data, that's optimized already (nobody gets information about all the objects, it's already divided to interesting collections).

So just data about position and orientations and their change so that client can represent object update in different frame rates, if object is constantly rotating, then we don't need to send data on every frame.

It's just like movie compression. We have film resolution about 1024x768, but it's not being saved as 2359296 bytes per frame. Some regions are constant, some regions are linearly moving. It's like my problem. Most of advices here are just like "please lower the resolution and you'll be happy" or "do not use 24-bit color" or "make 1 frame per second". So like I already have the task with high-quality videos, 24-bit color, high-speed - it's just an example.

The clients should have the same physics code as the server. Think of the clients like DirectX States are done where the GPU continues using the last State until it is changed. Send the player the initial velocities, pos, etc of the object and then let the clients figure out most of the res while sending sync updates on some time delay that is the longest time you can delay without screwing up what normal players see on screen. These numbers are all relative to your simulation, and will need to be tested to find.

*edit*

Just read your above post and it seems like you already know the answer.