Sign in to follow this  
doctorsixstring

Lockstep RTS: 90% done, 90% to go

Recommended Posts

The good news is that I have multiple instances of my game successfully running in lockstep across the loopback interface. Each instance runs in a small 400x300 window that allows me to issue commands from any player I want. I can select a few of one player's ships and give them a move order, and a very short time later the ships are moving on all the other player's screens. The bad news is that each game stalls fairly frequently. I'm still analyzing the data, but I'm having trouble coming up with any decent theories. The game instances just can't seem to communicate fast enough to keep ahead of the simulation updates. I'd like to hear any general advice or thoughts on what my problem might be. I doubt what I've said so far will be very useful, so I'll attempt to post a concise summary of my logic below. First, some background: My game is being developed in Python with PyGame and PyOpenGL. I'm using Python's socket module for my low-level communication. This will eventually be replaced with something like enet or RakNet, but for now the sockets allow me to easily send messages between players via TCP/IP. I'm working on a 1.6 ghz laptop. Running two instances of my app yield frame rates of roughly 80 FPS. Ping is usually around 10 ms, but I've had values as high as 20 ms across the loopback. The game will attempt to update the world every 50ms (i.e. 20 "ticks" per second). Player commands are scheduled two ticks in the future. Without further ado, here is my main loop with special emphasis on the lockstep networking code:
-------------------------------------

receive messages from other players over network

process local input and create game commands

sysTime = store local system time

if local input commands have been created:
	schedule commands in the local world simulation for world tick + 2
	send commands to other players

if player is host...
	lastAllReadyTick = last tick for which a "ready" message was received from all players
	if an "all ready" message wasn't sent for lastAllReadyTick...
		tickLength = calculate best tick length that can be acheived by each player (looks at pings and frame rates)
		send "all ready" message to all players.  The message includes the calculated tick length

if sysTime > nextTickTime:

	nextTickTime = sysTime + tickLength
	stalled = lastAllReadyTick < world tick

	if not stalled...
		if local player is a client and hasn't sent a "ready" message for world tick + 2...
			bestTickLength = highest average ping value to any other player OR local framelength (whichever is greater)
			send "ready" message to host

		update world simulation
		world tick += 1

render graphics
-------------------------------------
In case it isn't obvious from my pseudo-code, here are a few of my high-level thoughts: 1) Each player updates the world simulation at fixed intervals. 2) At the start of each tick, each client will send a "ready for tick X" message to the host. 3) When the host receives "ready for tick X" from all players, an "all ready for tick X" message will be sent to all other players. 4) The "lastAllReadyTick" variable stores the last tick for which an "all ready" message was sent or received. The game is stalled if the world simulation's tick exceeds this value. 5) The tickLength variable may change periodically, due to calculations by the host player's game. Changes to tick length will be when the next tick is scheduled. For example, the host player may schedule tick 7 immediately after setting tickLength to 55. Client #1 may not receive the updated tick length until tick 8, due to network latency. In that example, the host and client's tick lengths for tick 7 will be different. 6) The "world tick" variable is technically the simulation tick that will be updated next. Therefore, the game instance is considered "stalled" if lastAllReadyTick < world tick. If lastAllReadyTick == world tick, then the simulation still has one more tick to process. I apologize for the length of this post. Hopefully someone takes the time to read it and give me that small nugget of advice that will push me forward. I can even hope that this thread would eventually become a decent resource for future lockstep RTS developers. Thanks in advance, Mike

Share this post


Link to post
Share on other sites
So, are you basically counting the game as 'stalled' if you don't get a response within the 50ms? My concentration span's too short to read all your code and explanations to try and work out what you mean, I'm afraid.

How much data do you send each time? And have you tried "mySocket.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)"?

Share this post


Link to post
Share on other sites
If you're only sending small amounts of data relatively infrequently and using TCP, even with Nagle coalesing disabled you could be suffering from the way most stacks now utilise delayed ACKs. As insane as it sounds, see if the stalls happen less if you increase the amount of data you're sending.

Share this post


Link to post
Share on other sites
If you are running multiple games on a single CPU, scheduling (as in CPU/threads/processes) may come into effect. The Windows pre-emption quantum can be as big as 150 milliseconds, depending on system settings.

Try adding a Sleep(10) after processing each game tick, in each client. This will reduce frame rate, but will probably improve responsiveness.

Last, for play over the internet, a two-step delay between command and action is not enough for practical play. I would suggest counting on at least a 250 ms latency for robustness.

Share this post


Link to post
Share on other sites
It'd probably help, unless they both end up getting scheduled onto the same core. Do you have any problems if just two copies are playing? If you hook it up to a second computer, do you have any problem with two instances on the one and a third on the other?

Share this post


Link to post
Share on other sites
Quote:
Original post by Kylotan
You're running a PyGame app; are you emptying the event queue every single frame (as you should be)?


I believe so. At the start of the main loop, I have a large "for event in pygame.event.get():" loop. That should clear the event list, right?

I'm starting to think I have a fundamental flaw in my logic. The host instance on my laptop was stalling, even when the client instance was running on my desktop machine. Tick length was set to 100ms, and command delay was set to 5. Stalls seemed to occur less frequently, but they still happened fairly regularly.

Another idea I just had was to schedule ticks based on the scheduled tick time + tick length. Right now, I'm scheduling the next tick based on the current system time. If tick X was scheduled to occur at time 1000, it might not actually be processed until 1020, depending on how quickly the main loop is running. Tick X + 1 would then be scheduled for 1120, which significantly disrupts the "one tick every 100ms" rule. If I scheduled the next tick time for 1000 + 100, I would get a more consistent tick rate on all machines, which might help eliminate the stalls.

I didn't try padding my messages with extra data, so I'll try that tonight, too.

- Mike

Share this post


Link to post
Share on other sites
Quote:
Original post by doctorsixstring
I believe so. At the start of the main loop, I have a large "for event in pygame.event.get():" loop. That should clear the event list, right?


Actually, that isn't quite right. Here is my main loop:


while not quit:

timer.update()
input.update()

if net != None: # net object is None in single-player games
net.run()

for player in players:
player.update()

currentState.update()

scheduler.update()

console.update()

renderGraphics()



1) net.run() checks for incoming messages from the network.

2) currentState.update() loops through pygame.events.get().

3) scheduler.update() processes the lockstep logic and updates the world simulation.

Would the order of that logic matter? I'm not sure if there is a problem with doing local input processing between receiving network messages in net.run() and processing the lockstep logic in scheduler.update(). It all needs to processed either way, right?

Share this post


Link to post
Share on other sites
Quote:
Original post by doctorsixstring
Another idea I just had was to schedule ticks based on the scheduled tick time + tick length. Right now, I'm scheduling the next tick based on the current system time. If tick X was scheduled to occur at time 1000, it might not actually be processed until 1020, depending on how quickly the main loop is running. Tick X + 1 would then be scheduled for 1120, which significantly disrupts the "one tick every 100ms" rule. If I scheduled the next tick time for 1000 + 100, I would get a more consistent tick rate on all machines, which might help eliminate the stalls.


This has appeared to make a big difference. I've been testing on two seperate machines with 50ms ticks and a 5 command delay, and I've eliminated pretty much all stalling. A command delay of 2 shows minimal stalling. I currently have both tick length and command delay stored in a config file, and I'll probably keep it that way. Maybe I could even add a feature where the scheduler would automatically scale the command delay like it already scales the tick length.

I've still got some other work to do with my multiplayer code, but this has helped a lot. Thanks for all the ideas, guys!

- Mike

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this