Overview:
I’ve learned a tremendous amount about game development and architecture in the past 7 months at Elf Island (http://elfisland.com). I thought I’d share a bit about how our backend stuff works and our current scaling capacity, which we are refining. I’ll not delve into some of the “secrets” of Elf Island to keep this post at a reasonable length. I’ll visit the client at a later date.
We make heavy use of Amazon Web Services. All Elf Island servers run on Amazon EC2+Elastic IP. Since we’re cloudified and have a process in place, we can scale to handle tremendous load quickly. Here’s a bit of detail.
Elf Island requires the following servers to run:
- app server (n)
- smartfox socket server (n)
- db server (1+n)
- web server for assets like swfs, images etc. (n)
- load balancer
where (n) means there are n running servers and (1+n) means there’s a primary server with some sort of redundant backup. This means we need a minimum of 5 servers to run the game.
The crappy diagram below (sorry, an artist I am not) is an approximation of how things are setup:
App servers are used for data which does not need to be “live,” such as inventory, buddies, in game mail, game data, analytics services, and much more. All services are locked down via auth/roles and AOP. Some services are locked per IP and other safeguards. In game mail, incidentally, is delivered to proper recipients (unlike Twitter DMs O_o I kid, I kid <3 Twitter).
Smartfox is a socket server we use to maintain user presence, chat, and live state. It is useful because it allows developers to extend it via “extensions.” We have thrown out the bulk of the built in functionality and built our “live” presence from the ground up as our game requires much more in terms of moderation and safety than a typical MMO. Chat, location and presence, room join logic, etc. are all custom.
We use redundant MySQL dbs in a master-slave setup. Not much to say here aside from all realms share the same db so users can travel from server to server and retain everything about their character.
Apache servers host our assets. We’re looking into CDNs like Cloudfront but we push assets frequently so there are caching issues to work through and more pressing things to deal with right now. We’ll be visiting this in the near future.
We have a load balancer in front of our web servers and our app servers.
We also have at least one machine dedicated to monitoring, which sends alerts if bad things happen. Amazon SimpleDB (bulk put) is used to log high volume data like chat.
Our uptime has been amazing. In the past six months, we’ve only been down twice for about 15 minutes (aside from closing nightly) because a script failed after a faulty deployment. That was not related to application stability. Our users like that.
A realm is a single Elf Island “server,” from a user’s perspective. A realm is really made up of several machines. A realm is a dedicated smartfox server plus any number of shared app servers. Specifically, we can scale smartfox servers independent of app servers since there is not a 1-to-1 mapping of smartfox server to app server. Currently, users on one realm cannot communicate with one another but we’ve talked about it as well as sharding realms over multiple smartfox servers based on load. All in the future. For now, realms are independent except for state.
Login flow:
When a client logs in, the load balancer directs them to an app server. The app server authenticates the user and presents a dynamic list of realms. We can literally spin up and bring down realms on the fly. Users then choose one of the available realms, at which point they are authenticated with smartfox and brought into the Marketplace (a room) on the chosen realm.
At this point, information regarding users in the room is pushed to the client and style information for elves is loaded from smartfox and the app server. We load as little data as possible at any given time to minimize load. Once the style data is received, the client loads the appropriate assets from any of a number of web servers. Our clothing system is complex but provides great versatility. Users can style the way their elf looks beyond most other 2D MMOs. Since Elf Island is a kids game, we can’t have nekkid elves. As such, while the various assets are loaded, we obscure each elf’s on screen representation. Although the assets themselves are vector graphics, we render, composite and cache them as bitmaps client side for maximum performance. This does increase the game’s memory footprint but it’s not too bad. At this point, users can interact with each other as well as Elf Island.
Chat:
Elf Island users may only use words in an inclusive dictionary. Users are clever, though, so we use heuristics to catch things that are not allowed, such as the sharing of telephone numbers, etc. Although we filter chat on the client, even stricter safeguards are in place on the server. If a user were to somehow circumvent client-side filtering, the server would not only drop their message, but alert the moderation team.
Moderation Tools:
Moderators have several tools at their disposal. I won’t go into detail, but lets just say they can view game events in real time so children on Elf Island are safe. Curious? 