Today's service disruption

Post Reply
User avatar
Mithreas
Emeritus Admin
Emeritus Admin
Posts: 2555
Joined: Sat Sep 06, 2014 3:09 am

Today's service disruption

Post by Mithreas » Fri May 26, 2017 10:42 am

Hi folks,

We're coming back up, and will hopefully be stable.

The server ran out of disk space earlier, which unsurprisingly caused all sorts of problems. The most important is that a number of characters got corrupted. The combined wizardry of Scholar Midnight and myself (ok, mostly her) should have recovered everyone, but there are about 40 characters that were logged out during the disk space issue and which we've had to roll back to a backup copy...

...and unfortunately backups haven't been running correctly for the last week, so all these characters have been restored to their week-old versions. Apologies for this.

It's possible that there are a small number of characters who logged out during the minute this all kicked off, and which are still corrupt. If the server reliably crashes when you log into it, please send a PM to me, Scholar Midnight and the Active Admins group, and we'll fix you up. In the meantime, please don't keep trying to log in :-)

-Mith
xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3±1 emulated in Javascript on an Apple IIGS at a screen resolution of 1024x1.For security reasons, please leave caps lock on while browsing.

User avatar
Liareth
General Admin
General Admin
Posts: 1167
Joined: Mon Sep 08, 2014 8:25 am

Re: Today's service disruption

Post by Liareth » Fri May 26, 2017 12:38 pm

To follow from Mithreas' post ...

The backup process silently failed four days ago due to an issue with the server, and because I've been very busy with personal stuff, I didn't notice it as quickly as I should have. This means that the last backups we have available are from the 21st May at 4pm.

We've rolled back about 40 characters so far by automatic script, though we're aware that some have been missed, so we're currently investigating further.

Things that have been rolled back:

- Items in your character's inventory.
- EXP (normal and adventure).
- Some other things that are specific to your character file.

We can't do anything for 1) and 3), unfortunately, but we do have logging for when players obtain levels. Therefore, if you've lost significant EXP in the rollback, contact the DM team who will look up your last obtained level in the logs and give you experience to put you back to where you were.

Post Reply