Hi everyone. I'm writing a Twisted based daemon for a simple text-based multiplayer game. The game logic goes like this. The client sends a string in a particular format (command.arguments (also separated by period)) to the server. The server receives that string, split it, and run the command along with the arguments. I already made it working.

Some of the commands called (login for example) has to get data from database. I'm using Twistar to make things fancy. Here's where I found some difficulties. The function which calls the available commands returns the result of the called command. The problem is that I don't know how to make my login function to return the result of the query.

Simply, what I want : send strings (or list, or number, or whatever) gained from the database to the client.

What I get now : the server sends a deferred object or an empty string or list (I've tried some ways to send things to the client).

I know it sounds so newbie, but this is my first Twisted program, and I'm still learning, so please spare me.

Well, I think the scripts will make things clearer:

The server.py.

[spoiler]

from twisted.application import internet, service

import protocols, services


udp_port = 10000
multicast_addr = '234.0.0.1'

dbuser = 'root'
dbpass = 'password'

main_service = service.MultiService()

game_service = services.GameService(dbuser, dbpass)
game_service.setServiceParent(main_service)

udp_service = internet.MulticastServer(udp_port, protocols.MulticastGameProtocol(multicast_addr, game_service), listenMultiple=True)
udp_service.setServiceParent(main_service)

application = service.Application('Celestial Eclipse')

main_service.setServiceParent(application)
 

[/spoiler]

The protocols.py

[spoiler]

from twisted.internet.protocol import DatagramProtocol


class MulticastGameProtocol(DatagramProtocol):

    def __init__(self, multicast_addr, service):
        self.multicast_addr = multicast_addr
        self.service = service


    def startProtocol(self):
        self.transport.setTTL(5)
        self.transport.joinGroup(self.multicast_addr)

        
    def datagramReceived(self, datagram, address):
        print 'Datagram %s received from %s' % (repr(datagram), repr(address))
        if '.' not in datagram:
            return 'Invalid command format.'

        celcommand_name, arguments = datagram.split('.', 1)
            
        self.celcommandRequestReceived(celcommand_name, arguments, address)

        
    def celcommandRequestReceived(self, celcommand_name, arguments, address):
        command_result = self.doCommand(celcommand_name, arguments)
        
        if command_result is not None:
            self.transport.write(command_result, address)

            
    def doCommand(self, celcommand_name, arguments):
        submitted_command = getattr(self, 'celcommand_%s' % (celcommand_name), None)
        
        if submitted_command is None:
            return 'Invalid command.'
            
        try:
            #return 'COMMAND SUCCESS!!!'
            return repr(submitted_command(arguments))
        except:
            #return 'Command failed.'
            return repr(submitted_command(arguments))
            
    def celcommand_login(self, arguments):
        username, password = arguments.split('.', 1)
        return self.service.login(username, password)

[/spoiler]

The services.py

[spoiler]

from twisted.application import service
from twisted.python import log
from twisted.internet import defer

from twistar.dbobject import DBObject

class Users(DBObject):
    pass
    
class GameService(service.Service):

    def __init__(self, dbuser, dbpass):
        self.dbuser = dbuser
        self.dbpass = dbpass
        
    def startService(self):
        service.Service.startService(self)
        
        from twisted.enterprise import adbapi
        from twistar.registry import Registry

        Registry.DBPOOL = adbapi.ConnectionPool('MySQLdb', user='root', passwd='password', db='test')
        self.user = Users()
        
        log.msg('Made connection to database.')
        self.loginResult = None


    def getLogin(self, username, password):
        return self.user.find(where=['username="%s" AND password_hash="%s"' % (username, password)], limit=1)
        
    def login(self, username, password):
        #self.loginResult = []        
        self.d = self.getLogin(username, password)
        
        def checkLogin(query_result):
            if query_result is None:
                return 'Login failed!'
            else:
                print '%s logged in' % (query_result.username)
                return query_result.username
                
                #self.loginResult = query_result.username
                #self.loginResult.append(query_result)
        
        self.d.addCallback(checkLogin)
        
        #It has to return something, but I don't know what to return.

        #return self.loginResult

[/spoiler]

There's no complicated logic yet. I started writing this thing just three days ago.

The deferred does fire. It prints the "bla bla bla logged in" properly.

The client is just a simple client which sends 'login.Sky Warden.123'. The arguments will be separated by the '.' as well.

I think that's all. Maybe it's just me who don't fully understand deferreds, or it's just a logic hole. Or maybe both. XD

Thank you everyone.

In general, for TCP servers, each client has a socket that the server sends responses back to.

In a general communications crossbar, you will pass along a "client" object to each handler of a command.

This handler will then call some "enqueue message" function on that "client" object.

The client object in turn is responsible for taking those enqueued messages and turning them into byte streams you send back out over the socket.

Thanks. That makes sense. I was thinking to send the data over the socket via callback, but I'm stuck since the service is in a different class with the protocol.

Can you please give the example code to do this? It will help me a lot.

Where do you think the callback should come from?

Uhm... soon after the deferred is made?

Typically, you keep a queue of outgoing messages per client.

Once per network tick, you loop through all the clients, and bundle up all the messages for a client into a network packet, and send that.

So it's like we add new bytes of data to the queued message for each loop. It's like we append the new chunk of data to the a list of the message right? I think I understand that concept, but I guess I'm weak at coding it. I'm new to Twisted after all.

Can you please give me an example code? I'm not lazy or what, but that's the way I can understand things better. By learning the code.

Hmm. Actually, I think the problem is this: You don't want the deferred/callback to be scheduled by the thing that wants to send out the message. The thing that wants to send out the message should just enqueue it on the client-representing entity.

Kinda like that. Well, actually it's simpler. What I want is to make the login() function to return the data I get from the database. Just that. My problem is that I don't know how to do that. :lol:

The getLogin() needs to fetch data from the database, so it returns a deferred. Then I add a callback (checkLogin) to process the result when the deferred fires. Nah! The problem is here. The function which gets the result is the callback (the checkLogin), but the function which needs to return it is login(), the caller.

Sorry for the bold text. I just use that to mention that they're the important part.

By the way, I've tried to follow your advice about a client object handler. Sounds cool, but I don't know how to get the address when a connection is made (I'm sure it's in the startProtocol function). I know how to do it in TCP, but not in UDP.

Random question: are you the creator of that pymmo module?

hplus0603, on 27 May 2013 - 23:25, said:
You can't directly make the login() function return the data. You have to instead defer a function that gets called with the result of the login. If I remember correctly, you use reactor.callLater() for this: http://twistedmatrix.com/documents/12.3.0/core/howto/defer.html

Which functions I need to defer? The caller of the login() function (the celcommand_login())? So I need to fetch the result of that function after it's ready? And if it needs more than x seconds it will return nothing?

Uhm... code please? If you fix my code I can learn it quickly. I know it sounds demanding and lazy, but I'm not. I'm very eager to learn, and the best way I can learn logic is by reading code.

I really need to get this concept. I'm sure once I understand this problem I will be able to make an interesting progress.


I'm also trying to make another version (in case my logic is wrong in this version) with another logic flow. The difference is that I write the data to the socket via callbacks. The problem I get in this version is circular dependency. The DatagramProtocol class needs the Service class to call the login function in it, and the Service class needs the DatagramProtocol class to do the transport.write.

Oh my brain...

I also made the more ignorant version. Everything runs in one service. It's very very dirty, but kinda working. I still don't feel right about it though. I will only use it if the first version can't be helped.

hplus0603, on 27 May 2013 - 23:25, said:
For getting the source address, in UDP, there is only one client socket, and the source address is available on each incoming packet; you then have to do the mapping from address to client yourself, whereas TCP does that for you.

Oh, yes. I know that the datagram contains the address of the sender. I was wondering if we could get the address exactly when a connection is made, like in TCP. I forgot that UDP only has one socket.

hplus0603, on 27 May 2013 - 23:25, said:
And, yes, the 4-hour "pymmo" example was indeed something I did one late night :-)

That thing is amazing, sir, and you said you did it while learning a new language. Your pymmo really needs more publication.

By the way, why do I have to connect to the Internet to use UDP? If I'm not connected to the Internet it will say "no such devices" or something similar.

Well, apparently I made the second version (the one which writes data to the socket via callback) working. I solved that circular dependency problem by subclassing the protocol class. The subclass of the protocol class is a client object which handles requests like what you suggested, so I think it's done for now. :lol:

Thanks for the advice. :)