Scheduled Crash

Moderators: Forum Moderators, Active DMs, Contributors

Post Reply
Dr_Hazard89
Posts: 901
Joined: Wed Oct 08, 2014 9:06 am
Location: Australia

Scheduled Crash

Post by Dr_Hazard89 » Fri Sep 28, 2018 2:22 pm

The few weeks I've been back I've noticed (GMT+10) every day at 3:31pm and 11:31pm the server has a mini-crash/hiccup, where everyone is booted off or at least lagspikes very hard for the duration.

Not the worst thing in the world, but thought I'd point it out in case it means something important. I don't know.
Characters: Xun'sali (ACTIVE), Tianae Tymeaneldth (Rolled), Sylali (Rolled), Magpie (Rolled), Ker'uanna Tymeaneldth (Rolled), Lepota Poklona (Rolled), Andariel Bloodletter (Shelved), Tahl'tril Cyredrretyn (Rolled), L'omithiel (Shelved).

TimeAdept
Posts: 1336
Joined: Sun Aug 06, 2017 7:02 am

Re: Scheduled Crash

Post by TimeAdept » Sat Sep 29, 2018 5:34 am

Just happened again as of this post. Seems to happen every single day like clockwork about this time. Everyone lags and crashes out then logs back in, but the server itself doesn't crash.

strong yeet
Posts: 292
Joined: Wed May 02, 2018 9:09 pm

Re: Scheduled Crash

Post by strong yeet » Sat Sep 29, 2018 6:51 am

I'm noticing this too.

TimeAdept
Posts: 1336
Joined: Sun Aug 06, 2017 7:02 am

Re: Scheduled Crash

Post by TimeAdept » Sat Sep 29, 2018 8:58 pm

Happened again on C&P.

Griefmaker
Posts: 882
Joined: Mon Sep 08, 2014 8:33 pm
Location: California

Re: Scheduled Crash

Post by Griefmaker » Sat Sep 29, 2018 11:17 pm

It is almost like when one of the old servers would save to a backup or whatever and everyone would be kicked off at that time. We would always schedule our adventures around it. Not sure if that is the case now, but likely the admins know.

User avatar
Kreydis
Posts: 422
Joined: Sun Dec 21, 2014 3:42 am

Re: Scheduled Crash

Post by Kreydis » Sun Sep 30, 2018 1:05 am

For me it's every ##:58. Usually around 4 6 9 and 11. But it's random at times. But it's aways at the 58 minute mark.
It's a Dwarf, no it's a Dragon, no it's a Halfling! I think.

Dr_Hazard89
Posts: 901
Joined: Wed Oct 08, 2014 9:06 am
Location: Australia

Re: Scheduled Crash

Post by Dr_Hazard89 » Sun Sep 30, 2018 5:32 am

3:31, again. Weird that some have different times.

And this time it ate an essence! Ree.
Characters: Xun'sali (ACTIVE), Tianae Tymeaneldth (Rolled), Sylali (Rolled), Magpie (Rolled), Ker'uanna Tymeaneldth (Rolled), Lepota Poklona (Rolled), Andariel Bloodletter (Shelved), Tahl'tril Cyredrretyn (Rolled), L'omithiel (Shelved).

User avatar
LittleWeasel
Emeritus Admin
Emeritus Admin
Posts: 688
Joined: Sun Sep 07, 2014 4:21 pm
Location: Close by...
Contact:

Re: Scheduled Crash

Post by LittleWeasel » Sun Sep 30, 2018 6:35 am

I have to double check timings, but yes, these days the servers back up twice a day rather than once a day.
I thought the backupcrashbug was actually fixed a long time ago, before it got moved. Will have to check.

However, if it is just you and not all players at that particular time, I actually suspect that your isp might be running their updates and maintenance at those times.
Don't take Life too seriously - you'll never get out of it alive...
Mind over Matter... now that I don't have a mind, it doesn't matter...
Ware the Wrath of the Weasel:
*nibble*
*cluck*

User avatar
Blood on my Lips
Posts: 372
Joined: Tue May 23, 2017 7:03 pm

Re: Scheduled Crash

Post by Blood on my Lips » Sun Sep 30, 2018 4:36 pm

I have been in game with Time Adept when he experienced these occurrences and experienced the same thing. Multiple times.

TimeAdept
Posts: 1336
Joined: Sun Aug 06, 2017 7:02 am

Re: Scheduled Crash

Post by TimeAdept » Sun Sep 30, 2018 5:11 pm

LittleWeasel wrote:
Sun Sep 30, 2018 6:35 am
I have to double check timings, but yes, these days the servers back up twice a day rather than once a day.
I thought the backupcrashbug was actually fixed a long time ago, before it got moved. Will have to check.

However, if it is just you and not all players at that particular time, I actually suspect that your isp might be running their updates and maintenance at those times.
IT's been said several times in this topic it's the entire server. Whether you actually crash out or manage to stay in and then get a full minute's worth of activity seems random, since NWN has an acceptable level of lagand latency it will accept before you actually get disconnected, and the lag caused by whatever's happening seems to be just on the edge.

So people will be in game and suddenly there will be right about a solid minute of lag, followed by 45 instances of "has left as a player" and then people instantly rejoining because the server didn't go down.

Dr_Hazard89
Posts: 901
Joined: Wed Oct 08, 2014 9:06 am
Location: Australia

Re: Scheduled Crash

Post by Dr_Hazard89 » Mon Oct 01, 2018 1:33 pm

Another thing to note is that the website also becomes unresponsive while everyone is lagging out/crashing. Other websites work. Other things work. Pretty sure it's only Arelith doing this and not my ISP.
Characters: Xun'sali (ACTIVE), Tianae Tymeaneldth (Rolled), Sylali (Rolled), Magpie (Rolled), Ker'uanna Tymeaneldth (Rolled), Lepota Poklona (Rolled), Andariel Bloodletter (Shelved), Tahl'tril Cyredrretyn (Rolled), L'omithiel (Shelved).

User avatar
Red Ropes
Posts: 1008
Joined: Sun Feb 28, 2016 11:42 pm

Re: Scheduled Crash

Post by Red Ropes » Mon Oct 01, 2018 1:37 pm

Maybe the server is getting yeet'd on by blackmailers or something? I know some servers used to get hit by russian / chinese hackers to do `50 bux or else kiddo` sort of things back in the day.
🤡

TimeAdept
Posts: 1336
Joined: Sun Aug 06, 2017 7:02 am

Re: Scheduled Crash

Post by TimeAdept » Wed Oct 03, 2018 5:32 am

Dr_Hazard89 wrote:
Sun Sep 30, 2018 5:32 am
3:31, again. Weird that some have different times.

And this time it ate an essence! Ree.
Happened again, last nigh,t tonight, same time, 12:31 AM my time (CST)

User avatar
woodbreeze
Arelith Silver Supporter
Arelith Silver Supporter
Posts: 263
Joined: Wed Apr 12, 2017 1:03 pm
Location: EST

Re: Scheduled Crash

Post by woodbreeze » Wed Oct 03, 2018 5:44 am

I'm having this problem too!

User avatar
Kreydis
Posts: 422
Joined: Sun Dec 21, 2014 3:42 am

Re: Scheduled Crash

Post by Kreydis » Wed Oct 03, 2018 2:01 pm

LittleWeasel wrote:
Sun Sep 30, 2018 6:35 am
I have to double check timings, but yes, these days the servers back up twice a day rather than once a day.
I thought the backupcrashbug was actually fixed a long time ago, before it got moved. Will have to check.

However, if it is just you and not all players at that particular time, I actually suspect that your isp might be running their updates and maintenance at those times.
I would point out that this happens more then 2 times a day for me, so I'd rule out the servers being backed up.

Also if it's our ISPs causing an issue, I'd expect there to be lag on everything I'm using, not just NWN. This wasn't until like 2 months ago. What changed to cause people to start timing out?
It's a Dwarf, no it's a Dragon, no it's a Halfling! I think.

TimeAdept
Posts: 1336
Joined: Sun Aug 06, 2017 7:02 am

Re: Scheduled Crash

Post by TimeAdept » Fri Oct 05, 2018 5:33 am

Right on schedule

User avatar
Mithreas
Emeritus Admin
Emeritus Admin
Posts: 2555
Joined: Sat Sep 06, 2014 3:09 am

Re: Scheduled Crash

Post by Mithreas » Fri Oct 05, 2018 8:00 pm

Backups are scheduled for 08:25 and 16:25 UTC. It looks like the crash is observed about 6 minutes after the backup script starts.

The backup script does two meaningful operations - it backs up the MySQL databases, and then copies the servervault and DB backups over to an external server.

I think the most likely cause of the hang is the database backup script causing MySQL to pause processing other rquests, causing NWN to pause waiting for a response [it's single-threaded so this will pause all activity], meaning it misses heartbeats and causes clients to think the server has gone away, meaning you all disconnect.

The fix is probably to break the database backup into chunks. It's possible to do backups on a per table basis, meaning we could break the backup up into a number of smaller operations and reduce the "one big lag spike" effect. However, that means we'll need to remember to update the backup script whenever we add a table, which is not ideal. (We could probably do a separate schema-only backup just to make sure we can easily recover if someone misses a table, but the data would be gone).

I'll talk to LittleWeasel and Irongron when I get a moment.
xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3±1 emulated in Javascript on an Apple IIGS at a screen resolution of 1024x1.For security reasons, please leave caps lock on while browsing.

User avatar
MissEvelyn
Arelith Silver Supporter
Arelith Silver Supporter
Posts: 1584
Joined: Sun Jul 12, 2015 8:43 pm

Re: Scheduled Crash

Post by MissEvelyn » Fri Oct 05, 2018 8:03 pm

On Distant Shores, exactly 1.31 AM (Eastern US time) is when I get a long about 1 minute lag. I don't actually crash, however, but it's always on the dot.

Because I know when it's coming, I'm usually safe and prepared for it. But again, I've never crashed or timed out from it. It's merely just a prolonged lag time and then everything is fine after that.

I can't speak for the other servers however.
Last edited by MissEvelyn on Fri Oct 05, 2018 10:30 pm, edited 1 time in total.


Hinty
Posts: 327
Joined: Wed Aug 08, 2018 1:03 am

Re: Scheduled Crash

Post by Hinty » Fri Oct 05, 2018 9:20 pm

Don't remember it ever causing me to crash out. I have on occasion logged out under the mistaken impression that the server has crashed, but whenever I wait it out the server catches up again and carries on without issue.

dysmetria
Posts: 50
Joined: Sat Aug 04, 2018 3:17 am

Re: Scheduled Crash

Post by dysmetria » Sat Oct 13, 2018 1:09 am

LittleWeasel wrote:
Sun Sep 30, 2018 6:35 am
I have to double check timings, but yes, these days the servers back up twice a day rather than once a day.
I thought the backupcrashbug was actually fixed a long time ago, before it got moved. Will have to check.

However, if it is just you and not all players at that particular time, I actually suspect that your isp might be running their updates and maintenance at those times.
It isn't just them, at least not at the many times it has happened with me and two friends I have been playing with. We all keep lagging out at the same time. I'll try to track the times from now on.

It also rolls you back, but not your bank account, so when it happened an hour ago right after I withdrew everything I had, I logged back in to find all my gold missing from both my character and bank account.


Edit: Just happened to all three of us at :32 after. My character was again rolled back and lost thousands of gold. So it seems not only is the backupcrash bug back, but we aren't even being backed up before we crash.

When we log back in me and my friends are back to where we last entered the area we crashed in. So my guess is that is when our progress gets saved on our characters, during transitions.

I'm just surprised no one else is complaining about it since I've lost tens of thousands of gold twice in less than 12 hours because of it and it happens at least twice a day.

mf_hansen
Posts: 59
Joined: Fri May 15, 2015 6:17 am
Location: Denmark EU

Re: Scheduled Crash

Post by mf_hansen » Wed Oct 17, 2018 8:14 pm

I always -save right after banking transactions just to be safe, and barter with players instead of dropping items on ground/containers, but there no there is never a guarantee your character is 100% safe from disconnects/server crashes.

And yeah, it would be neat if database backups could somehow measure time spent and only run for so long before pausing, to let gameplay stay alive within a timeout limit, then continue on from where it stopped, and so forth until finished. Probably difficult to code so it can also account for new tables automatically, unless it somehow can get a runtime generated list of all tables to back up and store that list somewhere in its backup logic.
Send PM or find mf_hansen on IRC or Discord between 18:00-23:00 CET/CEST.

User avatar
triaddraykin
Arelith Silver Supporter
Arelith Silver Supporter
Posts: 735
Joined: Mon Sep 08, 2014 2:32 am

Re: Scheduled Crash

Post by triaddraykin » Sat Nov 24, 2018 6:37 am

Just resurrecting this thread to say it's still happening, just did.
Alia: The uncanny knack of Angela Amana to make the otherwise perpetually well-mannered girl lose her temper and be HORRIBLY ASHAMED afterwards.

Who on the Team to PM and When
Public Tower Discord
Tower Library
Scroll, Wand, & Potion Costs

User avatar
SteelsSweets
Arelith Silver Supporter
Arelith Silver Supporter
Posts: 441
Joined: Mon Sep 08, 2014 4:59 am
Location: Oooooooklahoma
Contact:

Re: Scheduled Crash

Post by SteelsSweets » Sun Dec 16, 2018 4:26 pm

A little threadomancy but it's still happening.

Can we get an update? Is this just a "perk" that we need to plan for? Sounded like a fix would be quite involved and could cause issues down the line.
Discord: SteelsSweets#4815

User avatar
MissEvelyn
Arelith Silver Supporter
Arelith Silver Supporter
Posts: 1584
Joined: Sun Jul 12, 2015 8:43 pm

Re: Scheduled Crash

Post by MissEvelyn » Sun Dec 16, 2018 10:25 pm

Would it be possible to have scheduled resets right before the backups? I imagine that would help on it.


User avatar
MalKalz
General Admin
General Admin
Posts: 3023
Joined: Sat Jun 10, 2017 10:33 pm

Re: Scheduled Crash

Post by MalKalz » Sun Dec 16, 2018 10:29 pm

It would not make a difference in the server hanging up.

The backup functions would still go through all the necessary backup procedures regardless.

Discord: @malkalz
Determine your Public CD Key here
Can't see your vault? Have you migrated your accounts? If you have tried, and still can't see them, message me.


Post Reply