Design patterns for servers with long life sockets?

Hello,

out of curiosity and personal interest I'm writing a simple
multiplayer game server that uses TCP.

Players will be able to log in with username and password, and with
each account one and only one "character" is associated. The game
itself will be very simple - that's not what I want to discuss.

I'm trying really hard to find an elegant structure for the whole
"connect", "log in", "start playing" procedure. Primarily it gets very
messy with the socket - character association and the messages that
need to be considered and potentially discarded, depending on "client
state". Multiple threads make it even worse.

Are there any useful design patterns out there for creating elegant,
robust and maintainable servers that have persisting sockets? What I
mean by persisting sockets is that the server does not function as a
pure "read, interpret, reply, close" service, but rather where sockets
are kept open for a long time and a continuous stream of messages are
sent back and forth. Usually this type of server also has several
states in which clients will be considered - depending on whether they
are logged in or not and so forth.


The current design:
* Server thread accepts socket
* Server thread dispatches this socket to a new client handler which
is enqueued into a thread pool for execution.
* Client handler thread reads from socket, parses text messages and
puts them in the "World message queue".
* World thread pumps the world message queue for messages.
* World thread calls a ClientState (a java enum with possible values
connected, logged_in, disconnected), that is a member of the client
handler object, which kind of filters the incoming messages. This enum
also has data that associates messages with code, String -> Method
basically.
* ClientState may or may not acknowledge the message. If it does, it
will call a function in the client handler.
* Client handler raises an event, and.. world object listens.
* I get confused and miserable.


As you can see it's a big mess. What I'm really trying to get at is a
data-oriented and elegant approach to client/character -> message ->
code association, with an easily maintainable structure with very
little code added for new message types and features.

Any good resources/books for this type of servers?

"magnus.wolffelt@gmail.com" <magnus.wolffelt@gmail.com> writes:
> Are there any useful design patterns out there for creating elegant,
> robust and maintainable servers that have persisting sockets? What I
> mean by persisting sockets is that the server does not function as a
> pure "read, interpret, reply, close" service, but rather where
> sockets are kept open for a long time and a continuous stream of
> messages are sent back and forth. Usually this type of server also
> has several states in which clients will be considered - depending
> on whether they are logged in or not and so forth.
>
> The current design:
> * Server thread accepts socket
> * Server thread dispatches this socket to a new client handler which
> is enqueued into a thread pool for execution.
> * Client handler thread reads from socket, parses text messages and
> puts them in the "World message queue".
> * World thread pumps the world message queue for messages.
> * World thread calls a ClientState (a java enum with possible values
> connected, logged_in, disconnected), that is a member of the client
> handler object, which kind of filters the incoming messages. This
> enum also has data that associates messages with code, String ->
> Method basically.
> * ClientState may or may not acknowledge the message. If it does, it
> will call a function in the client handler.
> * Client handler raises an event, and.. world object listens.
> * I get confused and miserable.

I'm going to try to beat Phlip to the punch and ask "Do you have
unit tests for that?"

You're getting confused and miserable because you're trying to
solve the whole problem at once. Do the bits you understand, then
look at it again. The first thing you need to do is accept a
connection from a client. Write a test client that connects to the
server, and make sure it fails. Now write a server that accepts the
connection. Your test passes.

Now write another client test that sends a login message. Watch
it fail. Modify the server until the test passes.

Next, run two clients at the same time. Watch the server fail to
handle one or the other. Modify it until the test passes (select() is
your friend here).

As you continue to write your tests and build your server, you'll
probably find that the Simplest Thing That Could Possibly Work (look
it up) will be to separate the processing from the connection
handling. Don't think of queues yet, think of a blackboard. Shared
memory is useful for that. Look at Jini and JavaSpaces, and steal
ideas if you don't have to use Java.

Finally, write some more tests, watch them fail, and then make
them pass.

Regards,

Patrick

------------------------------------------------------------------------
S P Engineering, Inc. | Large scale, mission-critical, distributed OO
| systems design and implementation.
pjm@spe.com | (C++, Java, Common Lisp, Jini, middleware, SOA)

Patrick May wrote:

> I'm going to try to beat Phlip to the punch and ask "Do you have
> unit tests for that?"

Honest thanks: I had already declined to answer due to inexperience with
long-term sockets. A MMORG, in theory, calls for a custom protocol on UDP, not
TCP, but then you get into hacking, security, and other fun murky middleware
issues that I have never actually done. I _try_ not to mention automated tests
if I don't have anything else to say...

The point of UDP is if your sockets are long-life but stateless, then they are
really short life. If the UDP packet does not arrive, then it does not, and life
goes on...

--
Phlip

Phlip <phlip2005@gmail.com> writes:
> Patrick May wrote:
>> I'm going to try to beat Phlip to the punch and ask "Do you
>> have unit tests for that?"
>
> Honest thanks: I had already declined to answer due to inexperience
> with long-term sockets. A MMORG, in theory, calls for a custom
> protocol on UDP, not TCP, but then you get into hacking, security,
> and other fun murky middleware issues that I have never actually
> done. I _try_ not to mention automated tests if I don't have
> anything else to say...

Now, now, is that really the STTCPW? ;-)

I do a lot of distributed computing work and I've seen people end
up building TCP over UDP, badly, more than once. My gut tells me that
you're correct, but I don't think it's a foregone conclusion.

> The point of UDP is if your sockets are long-life but stateless,
> then they are really short life. If the UDP packet does not arrive,
> then it does not, and life goes on...

Resiliency in the face of unreliable components is essential.

Regards,

Patrick

------------------------------------------------------------------------
S P Engineering, Inc. | Large scale, mission-critical, distributed OO
| systems design and implementation.
pjm@spe.com | (C++, Java, Common Lisp, Jini, middleware, SOA)

Patrick May wrote:

> Resiliency in the face of unreliable components is essential.

But that's what I meant: If one UDP does not get thru, the next one will, and
the state will get updated anyway. Doesn't streaming audio use UDP simply to
permit endlessly wide broadcasts without using a dedicated connection to every
recipient? Streaming MMORG graphics should work like that too, right?

And I might recall overhearing that some existing MMORG middleware stack used
UDP - but maybe I'm just mis-remembering.

--
Phlip

Phlip <phlip2005@gmail.com> writes:
> Patrick May wrote:
>> Resiliency in the face of unreliable components is essential.
>
> But that's what I meant: If one UDP does not get thru, the next one
> will, and the state will get updated anyway. Doesn't streaming audio
> use UDP simply to permit endlessly wide broadcasts without using a
> dedicated connection to every recipient? Streaming MMORG graphics
> should work like that too, right?
>
> And I might recall overhearing that some existing MMORG middleware
> stack used UDP - but maybe I'm just mis-remembering.

I was agreeing with you! Now you had to go and make it _violent_
agreement!

Regards,

Patrick

------------------------------------------------------------------------
S P Engineering, Inc. | Large scale, mission-critical, distributed OO
| systems design and implementation.
pjm@spe.com | (C++, Java, Common Lisp, Jini, middleware, SOA)

On Sat, 23 Aug 2008 16:05:39 -0700 (PDT), magnus.wolffelt@gmail.com wrote:

> I'm trying really hard to find an elegant structure for the whole
> "connect", "log in", "start playing" procedure. Primarily it gets very
> messy with the socket - character association and the messages that
> need to be considered and potentially discarded, depending on "client
> state". Multiple threads make it even worse.
>
> Are there any useful design patterns out there for creating elegant,
> robust and maintainable servers that have persisting sockets?

(I don't know how sockets can "persist". Do you mean to keep a dead socket
object in order to reuse?)

> The current design:
> * Server thread accepts socket
> * Server thread dispatches this socket to a new client handler which
> is enqueued into a thread pool for execution.
> * Client handler thread reads from socket, parses text messages and
> puts them in the "World message queue".

That is when the message is to change the state of the world. But I guess
that rather each client has some associated "agents" in the world to be
influenced by the message, and these agents in turn influence the world and
each other. When an agent receives a message it can handle it (change the
state of itself), respond to the client, when necessary (normally it would
be wasting resources if you are using TCP/IP), and then it can communicate
its new state with the world or other agents (over messages or any other
mechanism).

All in one, I think you have to separate communication with the clients and
agents, active objects of the world.

> * World thread pumps the world message queue for messages.
> * World thread calls a ClientState (a java enum with possible values
> connected, logged_in, disconnected), that is a member of the client
> handler object, which kind of filters the incoming messages. This enum
> also has data that associates messages with code, String -> Method
> basically.

As I said above, I don't see a reason why this need to be pumped through
the world thread. But if you want to, put a reference to the object
servicing the client to the message and call Handle_Me on it. You could use
rendezvous to avoid messages, but Java does not have them, AFAIK.

> As you can see it's a big mess.

Hmm, not very big, actually. You need a bit more design to restructure the
problem.

> What I'm really trying to get at is a
> data-oriented and elegant approach to client/character -> message ->
> code association, with an easily maintainable structure with very
> little code added for new message types and features.

> Any good resources/books for this type of servers?

I think any book on OO design would help. Look, you are talking about
"data-oriented" approach, that is a non-starter here!

(I think topmind might have a reserved opinion on the matters (:-))

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On Sat, 23 Aug 2008 17:56:16 -0700, Phlip wrote:

> But that's what I meant: If one UDP does not get thru, the next one will, and
> the state will get updated anyway.

Right, provided you are dealing only with the process variables, which are
continuous in their nature. Velocity, for example. This stops working when
you need to handle events and commands. Yet another issue is data
consistency. That is when a state is composed of other states. Consider an
double value of which the first word is lost, but the second was delivered.
And UDP not only does not warranty delivery, it also does not the order of.
This may have very nasty effects on the simulation, like artefacts, jerky
movements etc, even if nothing gets lost.

Actually the only use of UDP was for broadcasting. Luckily, we finally have
in-order delivery safe multicast protocols. So UDP may rest in peace.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Thanks for the replies. I do already have an automated test that does
the whole log in procedure.
I have actually had the server working quite well, even to the point
where after loggin in the client would receive a world definition (a
2d grid basically, think pacman).

Then I became dissatisfied with the structure, wanted more elegance...
and started messing about with queues and threads all over the place,
so now it's broken and I can't make up my mind on a design.
What I want to avoid is a ton of if/else if/else if/else if statements
for all the message handling and filtering..

> The point of UDP is if your sockets are long-life but stateless, then they are really short life. If the UDP packet does not arrive, then it does not, and life goes on...

Well, how does one reliably handle logging in and staying logged in
with UDP? Isn't that a sort of state?


> (I don't know how sockets can "persist". Do you mean to keep a dead socket
object in order to reuse?)

No. Consider a http server. It accepts a connection, reads a GET
command, builds a reply, sends the reply, closes the socket. This is
very simple and is easily programmed. But when you have a continuous
stream of messages back and forth, there is a need for more structure
to manage all the clients in an elegant fashion. I'm simply looking
for some design patterns that will properly constrain the code from
going messy and heterogenous.

On Sun, 24 Aug 2008 02:26:54 -0700 (PDT), magnus.wolffelt@gmail.com wrote:

> But when you have a continuous
> stream of messages back and forth, there is a need for more structure
> to manage all the clients in an elegant fashion.

That is rather the normal case. I don't see anything difficult in hading
it. You have some object responsible for a connection. Upon establishing a
connection, the listener creates the object and passes the socket to it.
When the connection is closed, the object dies and upon its finalization it
closes the socket handled by it. Depending on the communication mode, half-
vs. full-duplex you may have one of two threads associated with the object.

Well, in a full-blown middleware there also could be more complex things
like events schedulers, which ensure certain quality of service for the
subscribers, perform coalescing of events etc. But I doubt that this would
apply to your case.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de