Distributed MMORPG technology (just discussion about, not part of game project!)

Started by Kerame Pxel Nume, June 12, 2010, 07:06:35 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Kerame Pxel Nume

Quote from: bommel on June 12, 2010, 05:41:03 AM
Well, for a MMORPG we probably need some infrastructure which will cost money, depending on the popularity of the game one server won't be enough and this goes up to a few hundred bucks a month... but I won't generally exclude the possibility of making an mmorpg ^^

I have some servers at home and do some self hosting stuff but you really need a fast upload speed (at my parents we have just a crappy 3 mbit/s ADSL line with 50-70 kbyte/s upload. For serving webpages to a small amount of people this is sufficient but not for hosting a mmorpg ^^
This is the problem my engine design tackled: Its design is fully distributed, there are no centralized servers. Every participant's computer would provide computing power.

bommel

But don't you need accounting and ranking system? Somewhere centralized this data has to be stored, doesn't it? But as long as we have a small amount of players probably you won't need a lot of systems. Perhaps one or two dedicated machines? But this would probably go over 100$ again, at least if you're renting real (i.e. non-virtual) servers...

xMine

Other important questions when its a non-central system:
How do you "calculate" mobs? Then every client had to handle some of them and then send all this data
to clients, who are near these. Depending on how many users are around one enemy this could eat
up a lot of traffic. To solve this you could somehow set a bit in the "enemy position update" packet that
says, that the client has to forward this data to other clients around.

Then there is one important thing that might be bad: cheating. So any client had to check any packet
if it is okay or not. And when its not it should send a kick packet towards this client. But how can you be
sure that a cheated client obeys this? The other clients need to blacklist this ip (as well as nickname,...)
for some hours.

I think its an interesting idea, but I'm wondering, if this method has been used before in a MMO?
I know about some Ego-Shooter that does something like this and players says that it sometimes
laggs like hell...

EDIT:
I think at least one small server is needed anyway for ranking, account saving/handling, ...

Kerame Pxel Nume

Quote from: bommel on June 12, 2010, 07:12:56 AM
But don't you need accounting and ranking system?
Yes, but this can be distributed, too. The idea is, to have some machines being more authorative than others (the machines of moderators or so). Each moderator has a public key and every game state reported is peer reviewed by the moderator machines for plausibility, like that a player can't gain >x experience points within <T seconds, or that in a battle between player X (say a total newbie) and NPC Y (may be a Palulukan) the player wins.

Also content creation was distributed in a way similar to, or maybe actually by Git.

bommel

But you need to keep all this data coherent? I'm not that experienced with that stuff but don't you have a lot of synchronization overhead?

xMine

You always need at least one server to store all the data, but with this system
a small one is more than enough

Kerame Pxel Nume

Quote from: bommel on June 12, 2010, 07:39:22 AM
But you need to keep all this data coherent? I'm not that experienced with that stuff but don't you have a lot of synchronization overhead?
Well yes, but you won't have to synchronize to the whole world in realtime. Low Latency synchronization is only required between players which are in direct interaction. Such player clusters would vote a set trusted "accounting" nodes (each with it's own pubkey), based on a metric to choose those systems with good bandwidth and computing power, but most importantly its trustworthiness.

The outcome of each major battle is stored in a DHT to which every client has access. And with some latency each client checks if the reported outcome is consistent with his own records. If not it blames the accounting node responsible. But more important, each correct submission is signed of by the participating clients, giving the accounting node in question credit, so that it would be choosen over other nodes. However bad reports quickly drain this credibility, so cheating accounters are quickly disowned from the web of trust.

Quote from: xMine on June 12, 2010, 07:46:00 AM
You always need at least one server to store all the data, but with this system
a small one is more than enough
Actually you don't. Have a look in the way DHTs work.

xMine

But without any server, how do you get one ip (which should be enough) from a client?
DHT looks interesting, i think that's how wuala works. But when not a single client is online,
or not a single client with your account data (You can't store all data on every machine, that
database can grow huge), you're screwed...
Now I'm interested how you want to solve that...

Kerame Pxel Nume

Quote from: xMine on June 12, 2010, 07:55:44 AM
But without any server, how do you get one ip (which should be enough) from a client?
DHT looks interesting, i think that's how wuala works. But when not a single client is online,
or not a single client with your account data (You can't store all data on every machine, that
database can grow huge), you're screwed...
Now I'm interested how you want to solve that...
This is true, yes. But it's not a special kind of server that needs to stay up, but any kind of client. Of course we'd setup a set of primary seeding servers, but this don't do anything special. And while there's a sufficient large number of clients online, those servers actually aren't needed at all.

FYI: This is how Skype works: They don't have central servers, the whole Skype network is distributed over the clients, even user account data is stored therein.

bommel

Okay, so it's somewhat peer-to-peer like? I think I understand ^^

Kerame Pxel Nume

Quote from: bommel on June 12, 2010, 07:59:05 AM
Okay, so it's somewhat peer-to-peer like? I think I understand ^^
Yes. What I'm thinking is something like Magnet-Link-BitTorrent (where the whole content (terrain, assets, textures, models, etc.) date IS transferred by the very Magnet-Link-BitTorrent protocol)

Tsamsiyu92

xMine, why do you want the clients to share info? All goes through the server anyways, so this would only increase online traffic. One thing is important to consider. You don't need the position and all the information from a player on the other side of the world, so the server should only send player data of one player to players within sight.

Also, a peer-to-peer like system would require the players to open router ports, which is hell for those who aren't experienced with it. It could go without, but it would lag like hell. a CS:S player sends around 2kb/s, recieves 2kb/s. Think of a server having to deal with 1000 of them? You could of course try to send as little data as possible. Like your map (1 byte), position (2 byte) and how your character is animated, which is up to you to find out how.

I have really no knowledge about the coding of this, i have just made a simple game with awww.yoyogames.com[/url])] cheap game engine once.

xMine

Tsamsiyu92: Thanks for not reading the whole thread.
The idea comes from Kerame Pxel Nume and yes, i know about the port forwarding problem.

The main problem isn't the traffic (i think almost all gamers have a flatrate with unlimited traffic), it's
the needed bandwidth.
A Status Packet could look like this: User ID (4 Bytes), Position (3 x 4 Bytes = 12Bytes), Animation ID (2 Bytes).
Which is a total of 20 Bytes (Let's say 30 Bytes because of other information that needs to be send).
Then you only need to send this data to clients around you (100 ft. or something).
So, lets say there are 30 players around you. So you need 30 x 30 Bytes = 900 B/s = ~0,9kb/s which isn't much.
Even with a packet with 100 Bytes the client needs only to send 3kb/s. This could be even reduced when the
player clients around you are forwarding you packet. So it's not that huge.
We here in Germany have an average upload with 50kb/s minimum. So even when you'd send 20kb/s it shouldn't
be a big problem.

So the main problems:
1. Port Forwarding
2. Write some lib that can handle this
where only point 1 is important for normal users.
BTW: Somehow Skype manages that too and you don't need port forwarding.
When available the lib could use UPNP to interact with the router or some STUN technique

Kerame Pxel Nume

Quote from: Tsamsiyu92 on June 12, 2010, 09:06:53 AM
xMine, why do you want the clients to share info? All goes through the server anyways, so this would only increase online traffic. One thing is important to consider. You don't need the position and all the information from a player on the other side of the world, so the server should only send player data of one player to players within sight.

It was me who brought up this topic. And it was not about the actual game project, but the - now discarded - idea of a MMORPG and the engine concept I had in mind for that.

bommel

I think skype does this by cloaking the packet as something else (like HTTP). If I remember rightly ICQ does the same.

Kerame Pxel Nume

Quote from: xMine on June 12, 2010, 09:20:13 AM
The main problem isn't the traffic (i think almost all gamers have a flatrate with unlimited traffic), it's
the needed bandwidth.
A Status Packet could look like this: User ID (4 Bytes), Position (3 x 4 Bytes = 12Bytes), Animation ID (2 Bytes).
Which is a total of 20 Bytes (Let's say 30 Bytes because of other information that needs to be send).
Then you only need to send this data to clients around you (100 ft. or something).
So, lets say there are 30 players around you. So you need 30 x 30 Bytes = 900 B/s = ~0,9kb/s which isn't much.
Even with a packet with 100 Bytes the client needs only to send 3kb/s. This could be even reduced when the
player clients around you are forwarding you packet. So it's not that huge.
We here in Germany have an average upload with 50kb/s minimum. So even when you'd send 20kb/s it shouldn't
be a big problem.

I think it should be obvious that traffic should be compressed. All clients would support decompressing Deflate (ZLIB/GZIP), LZO, BZip2 and LZMA, whereas compression would be chosen by available CPU resources. Could even be done dynamically based on the number of spare CPU cycles until the next engine tick.

Decompressing either of them takes about the same time, but compression complexity varies strongly, with Deflate being quite easy, as is LZO, BZip2 modest and LZMA very expensive.

Quote from: xMine on June 12, 2010, 09:20:13 AM
So the main problems:
1. Port Forwarding
2. Write some lib that can handle this
where only point 1 is important for normal users.
BTW: Somehow Skype manages that too and you don't need port forwarding.
When available the lib could use UPNP to interact with the router or some STUN technique

http://samy.pl/pwnat/

Quote from: bommel on June 12, 2010, 09:23:20 AM
I think skype does this by cloaking the packet as something else (like HTTP). If I remember rightly ICQ does the same.
Nope, Skype uses some seriously tricky tunelling with a fully custom protocol, nobody except Skype can understand.

Eywayä lì'u

Oe prrkxentrrkrr ngeyä sa'nok!

GENERATION 18: The first time you see this, copy it into your sig on any forum and add 1 to the generation. Social experiment.

Kerame Pxel Nume

Quote from: Eywayä lì'u on June 12, 2010, 12:14:15 PM
again...

no online functionality
We're now in a separate thread. So anybody who'd like to discuss online gaming technology, be it distributed or centralized, feel free to do so. This thread is not about the actual game, but for those interested in the other technologies that exist.

Tsamsiyu92

xMine, I see how you think, but...

0,9 kb per sending, not second, it's not a lagfest either. Let's say you send that 4-5 times a second...

xMine

Quote from: Tsamsiyu92 on June 13, 2010, 03:26:55 PM
xMine, I see how you think, but...

0,9 kb per second, not second, it's not a lagfest either. Let's say you send that 4-5 times a second...

I've just corrected your mistakes...
1024 Bytes = 1kB. 1024kB = 1MB. ... and so on.
One packet would be around 30Bytes (more or less). 30Bytes multiplied by 30 clients around you is 900Bytes/Second.