MultiplayerDesign
From OpenCity
This article contains mails from the SDL mailing-list concerning the multi-player game desgin. Maybe it will be helpful to the future multi-player design of OpenCity.
Contents |
Leo M. Cabrera
On Sun, 2006-03-26 at 12:59 -0500, Leo M. Cabrera wrote: Hello everyone! :-D Alright, I'm making a multiplayer action game, you know, where people connect to a server and kill each other. :-) But I want to know how a good client-server should look like... I've never done network programming before. Should I do something like, the client moves, it sends the movement to the server, the server computes it, and it sends the updated character position to all of the clients... Or should I do something else? Thanks! ;-)
John Skaller
If you really want to do it properly you would use the equivalent of 'phase locking'. That's like driving a car.
Each client *estimates* what the other entities are doing, the server sends the real information (belatedly), and the client then corrects its estimation.
You cannot show the correct position at all times due to lag. This is not an artifact either -- it happens in reality too. One novelty you might try is give some graphical hint as to how much lag there is .. eg blur vision or something so the player can take the lag into account.
[of course when you see cows teleporting you know your ping is down .. but by then you're dead :]
Elden Armbrust
Personally, I would avoid visually representing each players lag in the game world. This can lead to higher ping players being targeted simply for that reason. Base your movements off of time, and only update their movement and actions when the client receives a packet. Example:
Server sends packets describing players. Clients receive packet and set state flags (such as velocity, direction, and actions such as running, walking, etc) Player A sees Player B running from left to right in front of him. Player A's client, based off of the player data for Player B (moving at n speed in i, j, k direction representing 3d spatial movement, running the "run" animation), Player B's position is estimated. 1 second passes, in which time Player B has stopped moving. Server sends packets describing players. Player A's client now adjusts, using an "easing" animation to go from the "run" animation to the "standing" animation.
Additionally, take multi threaded server operation into serious consideration. This is good for many reasons, but the most common of which is most likely network latency. A multi threaded server can easy cope with clients which are slow to respond, or which may become unreachable in the process of a game/connection. Also, if this is going to be a "massively multi player" game, consider using a master/slave setup for your server configuration (and subsequently, server programming). This allows what is considered "instancing" of various areas of the playing field, and can often tie in to a dynamic LOD system quite well. Also, it makes the server load for the primary server much lower overall, since it has but one job: handling network traffic.
One more thing to consider is network bandwidth. While clients are often capable of downloading data in speeds in excess of 5Mbit, not all servers or clients are capable of such speeds. Aim for a lowest common denominator. That will allow more clients at a lower ping rate overall. A good estimation from my experience is no more than 20 kilobytes per second. This does rule out dial up users, but opens the door to almost any and all broadband users. If you do aim for dial up users, remember that while 56k modem users are capable of downloading at speeds of upwards of 5.5kilobytes per second, their upload speed can and will be greatly effected by such. In this case, aim for no more than 3 kilobytes per second for each upload and download. This can be easily accomplished with packet scheduling (mind you, I'm not referring to OS specific packet scheduling but scheduling built into your application.).
Hope that helps you on your way.
Leo M. Cabrera
Oohh... So I use your example, and make a multithreaded server, with one thread for each player? Is that what you meant? I'll try that, thanks! ;-)
Elden Armbrust
That's the model I use (for the most part). Just in case you were wondering why, I can explain: Say there are 16 players playing a game, connected to a single threaded server application. Player 9 suddenly loses power at his home/apartment/etc. Due to the nature of TCP/IP (which is the norm, and what I assume you'll be using), when the server attempts to send a packet to the client, it will try to get an ACK packet back. However, due to the client no longer being reachable, the program will wait until a timeout is reached (or in the worst case scenario, ndefinitely) to resume operation. However, the remaining players will ALSO have to wait the same length of time to receive data packets from the server, since it cannot do anything else besides wait for that client which is no longer reachable. In addition to being able to manage your connections on a per thread basis, you eliminate the need to manage the connections by priority. This way, you can simply go about sending and receiving data on a per thread basis nearly without worry as to what the client is doing. Depending on the threading package/API you'll be using you may have the ability to manage errant threads nicely. (i.e. Kill a thread that is no longer responding due to being hung/blocked/etc) Oh yes, and one other nice little tidbit of information: When running an application such as that I've described above you should see a marked and in some cases impressive increase in speed and/or efficiency when running on a 64 bit processor (especially those with HT or similar technology).
Perhaps I'll write up a "network game theory" paper and link it in a post here or something (or even in the message itself).
Good luck
Jeff Jackowski
If that ever happens, it is with blocking TCP sockets. Using non-blocking sockets avoids the problem, and using UDP eliminates the ACK. UDP provides better performance than TCP, so many commercial games use UDP rather than TCP.
Also, having a kernel thread for each player doesn't scale well. MMO's will use a kernel thread for some number of players. Having 10,000+ kernel threads can keep the kernel pretty busy switching from one thread to another. For a few players, like 32, it isn't a big deal, but it'll force the threads to properly synchronize access to data shared between the threads to avoid race conditions and inconsistant data.
Using select() with non-blocking TCP, or a higher performace platform specific method, allows one thread to handle a number of clients. With UDP, there is no way to tell which client's data will be read on the next recv() call so a single-threaded server is the simplest solution.
John Skaller
Thanks you for the free advertisement for Felix! I couldn't have stated this better.
Felix threads scale: several million is no problem on an average PC. They're synchronous, so generally don't require locks: synchronisation is automatic and handled by channels. Switching is basically just a single C++ virtual function call, so it's quite quick.
Felix also does non-blocking async socket I/O behind the scenes automatically using the fastest available event source for your platform: IO completion ports for Windows, epoll on Linux, Kqueue on BSD and OSX, falling back to select() as a last resort.
This is all transparent to the application programmer.
The current library only runs TCP (guess I'll have to hassle the async I/O developer to put UDP in .. :)
A binding to SDL and OpenGL is part of the core system. Bindings to any C or C++ you like are easily created directly in the language, and rarely require any executable glue logic, since the system uses the same object model as C/C++ (and in fact generates C++).
Pre-emptive threading is also supported if you really need it. However most people use pthreads for control inversion and to convert blocking I/O to non-blocking I/O neither of which actually require concurrency, and both of which are supported directly in the core system.
Bugs and problems are fixed almost as fast as with SDL. [Give me a break! Sam fixes your bug *before* you report it, that's hard to compete with :]
Elden Armbrust
That is true that the blocking sockets are the cause for that, but I can't agree that UDP gives a *better* performance. A faster performance, almost definitely. An application using UDP cannot know whether data sent to a peer application is received by the other end or not. Additionally, for those that reach the destination, there is no guarantee on the ordering of the data, and the receiver may get duplicated copies of the same data. This can lead to congestion, or even "flooding" of the client system if the upstream of the server is significantly large enough. (Which is not uncommon with game servers)
Hence the reason i stated that I use that model mostly, and used an example of 16 players rather than 16,000. Also, 64 bit CPUs are capable of an exceptionally larger number of threads than a standard 32 bit CPU, which was why I mentioned them.
UDP (in my opinion) isn't exceptionally well suited to time critical applications such as games. Granted, some developers feel differently. However, in a game where you want to be certain the data is arriving the way you want it to, TCP is ideal. Dan Kegle has a great page describing different setups for large servers. While the site is a bit older now, he keeps it up to date and the information is still valid for larger server applications. If the server is to be an MMO style server, it's definitely worth checking out. The site is at http://www.kegel.com/c10k.html
Leo M. Cabrera
pthreads? Aren't those Linux-only? :-P (I'm using SDL threads) But anyways, I'll only allow about 10 to 20 players per server, so I can allow one thread for each player... And... I would really like for dial-up users to be able to play... I'll be sending and receiving the following struct to and from the users (in the thread):
struct Player
{
// Player's rotation, X-position, and Y-position
int Rotation, xPos, yPos;
// Person's name
char Name[NAMELENGTH];
// Team: true = red, false = blue
bool Team;
// Player's IPv4 address (text representation)
char IPAddress[16];
// IP address
IPAddress IP;
// Socket descriptor
TCPSocket SockDesc;
};
How much is that? 49 bytes (plus the last 2 variables)? Will a dial-up connection handle it? Also... For the threads, what processor speed and/or megabytes of RAM will the server need? I would like it if my old IBM comp (350MHz P2, 192 MB RAM) was the server... :-) Thanks you all! ;-)
Jeff Jackowski
No. The p stands for POSIX, and there is a Win32 implementation. Still, unless there is a reason not to, you should stick with SDL threads.
Probably more when you account for padding. Sort the elements from largest to smallest in the struct to limit the padding and save memory. Also, by writting each element to the buffer to send instead of the whole struct, you can avoid sending a struct with padding over the network. This also increases the portability of the code because different compilers and different platforms may align the data differently (it's 4-byte aligned on most systems today, but a 64-bit system may use 8-byte alignment). By writting each element yourself, you can also avoid writting data that seldom changes except for when it does change, like the player's name. Plus, you can limit the bytes used for a string, like the name, to strlen(string) + 1 bytes (less if you get fancy about it, but there isn't a great need).
That depends on how often you send updates. Also plan on a 100+ms latency introduced by the modem.
It depends on what your server is doing. If it does little more than repeat data to all the clients, than the old system you describe can handle it. Given that struct, I doubt your server will need to do so much that the old system would have trouble.
Leo M. Cabrera
Oh crap! Yeah, I forgot about the portability of structs... I now remember from when I learned about binary files... :-P
Does that mean that I have to wait 100ms at the beginning of the send/receive function thread?
I also have a question now... How do I send the data to the clients? I was thinking of sending an integer saying how much players it's going to receive, and then send them... Is that alright? Thanks! ;-)
PS: I'm so happy about this game! I hope it comes out well... :-)
Elden Armbrust
Well, 49 bytes is quite small...however...
You're not sending any animation data (if there is any). Also, I can't seem to grasp your reasoning for sending the IP adress every update, let alone more than once. Once you have the IP in memory, it shouldn't be necessary to send it over the network any longer unless it changes mid-game. If that happens, the client will most likely lose connectivity anyways. Also, the team (as far as I know, I'm not sure about your setup) shouldn't need to be sent each update either. If you were to leave out the IP addresses, the team boolean, and the TCPSocket type, replacing them with a single char as an identifier, you'd be left with a structure (not including padding) of 4 + 4 + 4 + NAMELENGTH
So if NAMELENGTH was 32 bytes, that would leave you at 44 bytes per packet. A 56k modem client could handle ~90 updates per second (plus a little overhead for padding, etc) With a proper scheduling setup, you could update 50-100 milliseconds, which would put you at 440 to 880 bytes per second (not including overhead) per player. It's highly likely that if you program carefully (i.e. being very careful to avoid even the smallest memory leak), your 350mhz machine could potentially handle 32 players 24/7 for quite a while. If the internet connection is fast enough, that is. For a 32 player game you would need ~880 * 32 upstream and downstream, which would be approximately 29kilobytes per second downstream, (~880 * 31) * 32, or ~870 kilobytes per second upstream (31 other players data sent to each of the 32 players). To reduce the amount of upstream bandwidth you'd need, you could update less, and reduce the size of the struct.
I would strongly recommend against sending a a character string with the players name during updates. At a 500ms update cycle and a 12 byte structure (dropping the name) you can realize an server upstream requirement of only ~24 kilobytes per second for a 32 player game. This is why I made such an issue of a good scheduling system for your data transmission. The slightest miscalculation can lead to poor network performance. Your current struct (at 49 bytes PLUS the last 2 variables) will most likely create horrible lag if hosted on anything but a 10 Megabit connection. Another thing to consider is using a char*3 to represent the rotation of the character, and converting the resulting data to degrees for use in your program. That will shave 1 byte from the structure every update.
I hope that helps but things in a bit of perspective. Please, if my math is way off on anything, let me know. Just remember that most everything is an estimation and not a hard number. :) Good luck
Jeff Jackowski
No, I just ment that your game should work OK with the greater latency. I can send data across the US in about 100ms, or across town with a modem in 100 to 150ms.
Since it seems like your trying to keep the networking part pretty simple, I shouldn't have made an issue out of it. Worry about it after you've got the networking code running and the latency is a problem. If it isn't a problem, don't worry :-)
That is certainly the easiest way. Even better is trying to send only data that has changed. If the gainularity there is all the info on one player then only when a player does something will the data be sent. That'll help support modem users.
To support that scheme, you may need to tell clients about new players entering the game and existing ones leaving the game. Then it can be useful to send messages. Based on the message type, the client or server will process the data differently. Then you can have connect, disconnect, and player update messages. It's pretty common to use some sort of message system rather than sending data for all players to all clients every time through some update loop.
Good luck, and have fun with it!
John Skaller
P for 'pre-emptive' :)
I played D2 for years on a 32K dial up connection with 500 ms latency (due to server being half way around the world). PvM was quite playable -- not crash hot for PvP though.
The thing is a 56K dialup running 56K upload is the same speed as 256K download ADSL. So I guess dialup will be fine for the clients at least.
Rhythmic Fistman
The current library only runs TCP (guess I'll have to hassle the async I/O developer to put UDP in .. :)
- I wrote some posix style stuff to do that (no win32 yet), but haven't yet tried it out with the demuxers. It's one of the many felix add-ons that I've got on the boil, including posix async io (aio_), epoll demuxer and even an sdl_net demuxer(lazy man's way of getting a classic mac/OpenTransport socket impl.) - a proposito of this, sdl_net's SDLNet_CheckSockets only checks for read-readiness and not write. What's with that?
Pre-emptive threading is also supported if you really need it. However most people use pthreads for control version and to convert blocking I/O to non-blocking I/O neither of which actually require concurrency, and both of which are supported directly in the core system.
- It's felix's reason for existing. However, if you're not feeling adventurous enough to code in a great new language and you're stuck in windows (>=nt4) and don't want to worry about concurrency, you can use fibers (co-operative threads). Nice to see that someone's finally wrapped up the old longjmp/setjmp/jmpbuf-fiddle dance.
Leo M. Cabrera
Yeah, alright, so all I'll send is 33 bytes, the rotation, x/y axes, name, and team flag. (I'll have to send some more data later on, such as being on a vehicle or not, weapon, etc... :-P) But, how do I make it only send data when it has changed? Do I make the client send a boolean flag before anything else, and if it's false, then quit receiving data from that user and go to the next one? Thanks you all! ;-)
