Aug 11, 2016 | Cloudflare
Okay, so things are almost settling down now, time to recap what the heck happened.
As we mentioned, back in January, we started hosting a new game on our server called Avabur. It's become pretty popular, but all hasn't been perfect. There have been two instances of troublemakers attempting to take the server down for everyone, across all our sites. In bird culture, this is considered a dick move
. The first one was not enough to take extreme measures, but the latest one caused some severe issues. Traffic was saturating our network link
, and without action, would cause a server bill in the thousands of dollars for the month.
Our solution was simple: the server did not move, but we placed a service called Cloudflare between the server and the internet. Cloudflare takes our incoming connections and filters them, removing any malicious traffic. Cloudflare then passes the "clean" connections to us. The server was then configured to delete any traffic not coming in via this new filtered connection.
Obviously, things are slightly more complicated than this sounds, and this has consumed the past 3 days. fury and andy have both been working on fixing all the issues, but there are still more problems ahead of us than behind. Keeping the sites online was the first priority, and we'll restore 100% functionality in the coming days and weeks.
The reset date may be affected by these events, as we want to have something ready for the new round. All non-essential work has been put on hold until the connectivity issues are resolved.
May 21, 2016 | New Round, New Problems (Age 23)
The reset, some of you may have noticed, had some interesting quirks this time around – looks like our restoration from backup restored a few old accounts that lurked under the surface until we tried posting the age-end stats! Weeding them out ended up being somewhat impossible, so they'll stay there for the time being.
Swiftly moving on to new things...
- Mercenaries now require upkeep, the cost will be deducted from your TBG at every turn (10 Gold for Attack and Defense Mercenaries, 5 Gold for Untrained Mercenaries)
- Elves now have a 20% chance to steal 10% of what they've just sabotaged!
Problems? Message andyt683
if you find something that doesn't work.
Apr 7, 2016 | Anatomy Of A Disaster
Yesterday morning, we had a component failure in our main database disk, making the entire server unresponsive. This was unexpected, as the drives were mostly new. All data on the drives was presumably lost, though we are exploring options to get it back. Attempts to swap to our backup server failed due to a strict security setting, preventing most browsers from viewing the alternate site. We are back up and running on the original server, although with a fair amount of data lost, and reduced performance. We've restored as much data as we had backed up, though more data may be restored down the road.
Going forward, we've ordered two new SSDs, and will be installing them in a RAID setup to add additional redundancy. Backup scripts are also far more comprehensive, as well as additional offsite locations for these backups. Full performance restoration will be completed within a week or two, barring any more unforeseen issues.