Scheduled Crash
Moderators: Forum Moderators, Active DMs, Contributors
-
- Posts: 901
- Joined: Wed Oct 08, 2014 9:06 am
- Location: Australia
Scheduled Crash
The few weeks I've been back I've noticed (GMT+10) every day at 3:31pm and 11:31pm the server has a mini-crash/hiccup, where everyone is booted off or at least lagspikes very hard for the duration.
Not the worst thing in the world, but thought I'd point it out in case it means something important. I don't know.
Not the worst thing in the world, but thought I'd point it out in case it means something important. I don't know.
Characters: Xun'sali (ACTIVE), Tianae Tymeaneldth (Rolled), Sylali (Rolled), Magpie (Rolled), Ker'uanna Tymeaneldth (Rolled), Lepota Poklona (Rolled), Andariel Bloodletter (Shelved), Tahl'tril Cyredrretyn (Rolled), L'omithiel (Shelved).
Re: Scheduled Crash
Just happened again as of this post. Seems to happen every single day like clockwork about this time. Everyone lags and crashes out then logs back in, but the server itself doesn't crash.
-
- Posts: 292
- Joined: Wed May 02, 2018 9:09 pm
Re: Scheduled Crash
I'm noticing this too.
Re: Scheduled Crash
Happened again on C&P.
-
- Posts: 883
- Joined: Mon Sep 08, 2014 8:33 pm
- Location: California
Re: Scheduled Crash
It is almost like when one of the old servers would save to a backup or whatever and everyone would be kicked off at that time. We would always schedule our adventures around it. Not sure if that is the case now, but likely the admins know.
Re: Scheduled Crash
For me it's every ##:58. Usually around 4 6 9 and 11. But it's random at times. But it's aways at the 58 minute mark.
It's a Dwarf, no it's a Dragon, no it's a Halfling! I think.
-
- Posts: 901
- Joined: Wed Oct 08, 2014 9:06 am
- Location: Australia
Re: Scheduled Crash
3:31, again. Weird that some have different times.
And this time it ate an essence! Ree.
And this time it ate an essence! Ree.
Characters: Xun'sali (ACTIVE), Tianae Tymeaneldth (Rolled), Sylali (Rolled), Magpie (Rolled), Ker'uanna Tymeaneldth (Rolled), Lepota Poklona (Rolled), Andariel Bloodletter (Shelved), Tahl'tril Cyredrretyn (Rolled), L'omithiel (Shelved).
-
- Emeritus Admin
- Posts: 688
- Joined: Sun Sep 07, 2014 4:21 pm
- Location: Close by...
- Contact:
Re: Scheduled Crash
I have to double check timings, but yes, these days the servers back up twice a day rather than once a day.
I thought the backupcrashbug was actually fixed a long time ago, before it got moved. Will have to check.
However, if it is just you and not all players at that particular time, I actually suspect that your isp might be running their updates and maintenance at those times.
I thought the backupcrashbug was actually fixed a long time ago, before it got moved. Will have to check.
However, if it is just you and not all players at that particular time, I actually suspect that your isp might be running their updates and maintenance at those times.
Don't take Life too seriously - you'll never get out of it alive...
Mind over Matter... now that I don't have a mind, it doesn't matter...
Ware the Wrath of the Weasel: *nibble*
*cluck*
Mind over Matter... now that I don't have a mind, it doesn't matter...
Ware the Wrath of the Weasel: *nibble*
*cluck*
-
- Posts: 372
- Joined: Tue May 23, 2017 7:03 pm
Re: Scheduled Crash
I have been in game with Time Adept when he experienced these occurrences and experienced the same thing. Multiple times.
Re: Scheduled Crash
IT's been said several times in this topic it's the entire server. Whether you actually crash out or manage to stay in and then get a full minute's worth of activity seems random, since NWN has an acceptable level of lagand latency it will accept before you actually get disconnected, and the lag caused by whatever's happening seems to be just on the edge.LittleWeasel wrote: ↑Sun Sep 30, 2018 6:35 amI have to double check timings, but yes, these days the servers back up twice a day rather than once a day.
I thought the backupcrashbug was actually fixed a long time ago, before it got moved. Will have to check.
However, if it is just you and not all players at that particular time, I actually suspect that your isp might be running their updates and maintenance at those times.
So people will be in game and suddenly there will be right about a solid minute of lag, followed by 45 instances of "has left as a player" and then people instantly rejoining because the server didn't go down.
-
- Posts: 901
- Joined: Wed Oct 08, 2014 9:06 am
- Location: Australia
Re: Scheduled Crash
Another thing to note is that the website also becomes unresponsive while everyone is lagging out/crashing. Other websites work. Other things work. Pretty sure it's only Arelith doing this and not my ISP.
Characters: Xun'sali (ACTIVE), Tianae Tymeaneldth (Rolled), Sylali (Rolled), Magpie (Rolled), Ker'uanna Tymeaneldth (Rolled), Lepota Poklona (Rolled), Andariel Bloodletter (Shelved), Tahl'tril Cyredrretyn (Rolled), L'omithiel (Shelved).
Re: Scheduled Crash
Maybe the server is getting yeet'd on by blackmailers or something? I know some servers used to get hit by russian / chinese hackers to do `50 bux or else kiddo` sort of things back in the day.
Re: Scheduled Crash
Happened again, last nigh,t tonight, same time, 12:31 AM my time (CST)Dr_Hazard89 wrote: ↑Sun Sep 30, 2018 5:32 am3:31, again. Weird that some have different times.
And this time it ate an essence! Ree.
-
- Arelith Silver Supporter
- Posts: 263
- Joined: Wed Apr 12, 2017 1:03 pm
- Location: EST
Re: Scheduled Crash
I'm having this problem too!
Re: Scheduled Crash
I would point out that this happens more then 2 times a day for me, so I'd rule out the servers being backed up.LittleWeasel wrote: ↑Sun Sep 30, 2018 6:35 amI have to double check timings, but yes, these days the servers back up twice a day rather than once a day.
I thought the backupcrashbug was actually fixed a long time ago, before it got moved. Will have to check.
However, if it is just you and not all players at that particular time, I actually suspect that your isp might be running their updates and maintenance at those times.
Also if it's our ISPs causing an issue, I'd expect there to be lag on everything I'm using, not just NWN. This wasn't until like 2 months ago. What changed to cause people to start timing out?
It's a Dwarf, no it's a Dragon, no it's a Halfling! I think.
Re: Scheduled Crash
Backups are scheduled for 08:25 and 16:25 UTC. It looks like the crash is observed about 6 minutes after the backup script starts.
The backup script does two meaningful operations - it backs up the MySQL databases, and then copies the servervault and DB backups over to an external server.
I think the most likely cause of the hang is the database backup script causing MySQL to pause processing other rquests, causing NWN to pause waiting for a response [it's single-threaded so this will pause all activity], meaning it misses heartbeats and causes clients to think the server has gone away, meaning you all disconnect.
The fix is probably to break the database backup into chunks. It's possible to do backups on a per table basis, meaning we could break the backup up into a number of smaller operations and reduce the "one big lag spike" effect. However, that means we'll need to remember to update the backup script whenever we add a table, which is not ideal. (We could probably do a separate schema-only backup just to make sure we can easily recover if someone misses a table, but the data would be gone).
I'll talk to LittleWeasel and Irongron when I get a moment.
The backup script does two meaningful operations - it backs up the MySQL databases, and then copies the servervault and DB backups over to an external server.
I think the most likely cause of the hang is the database backup script causing MySQL to pause processing other rquests, causing NWN to pause waiting for a response [it's single-threaded so this will pause all activity], meaning it misses heartbeats and causes clients to think the server has gone away, meaning you all disconnect.
The fix is probably to break the database backup into chunks. It's possible to do backups on a per table basis, meaning we could break the backup up into a number of smaller operations and reduce the "one big lag spike" effect. However, that means we'll need to remember to update the backup script whenever we add a table, which is not ideal. (We could probably do a separate schema-only backup just to make sure we can easily recover if someone misses a table, but the data would be gone).
I'll talk to LittleWeasel and Irongron when I get a moment.
xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3±1 emulated in Javascript on an Apple IIGS at a screen resolution of 1024x1.For security reasons, please leave caps lock on while browsing.
-
- Arelith Silver Supporter
- Posts: 1584
- Joined: Sun Jul 12, 2015 8:43 pm
Re: Scheduled Crash
On Distant Shores, exactly 1.31 AM (Eastern US time) is when I get a long about 1 minute lag. I don't actually crash, however, but it's always on the dot.
Because I know when it's coming, I'm usually safe and prepared for it. But again, I've never crashed or timed out from it. It's merely just a prolonged lag time and then everything is fine after that.
I can't speak for the other servers however.
Because I know when it's coming, I'm usually safe and prepared for it. But again, I've never crashed or timed out from it. It's merely just a prolonged lag time and then everything is fine after that.
I can't speak for the other servers however.
Last edited by MissEvelyn on Fri Oct 05, 2018 10:30 pm, edited 1 time in total.
Re: Scheduled Crash
Don't remember it ever causing me to crash out. I have on occasion logged out under the mistaken impression that the server has crashed, but whenever I wait it out the server catches up again and carries on without issue.
Re: Scheduled Crash
It isn't just them, at least not at the many times it has happened with me and two friends I have been playing with. We all keep lagging out at the same time. I'll try to track the times from now on.LittleWeasel wrote: ↑Sun Sep 30, 2018 6:35 amI have to double check timings, but yes, these days the servers back up twice a day rather than once a day.
I thought the backupcrashbug was actually fixed a long time ago, before it got moved. Will have to check.
However, if it is just you and not all players at that particular time, I actually suspect that your isp might be running their updates and maintenance at those times.
It also rolls you back, but not your bank account, so when it happened an hour ago right after I withdrew everything I had, I logged back in to find all my gold missing from both my character and bank account.
Edit: Just happened to all three of us at :32 after. My character was again rolled back and lost thousands of gold. So it seems not only is the backupcrash bug back, but we aren't even being backed up before we crash.
When we log back in me and my friends are back to where we last entered the area we crashed in. So my guess is that is when our progress gets saved on our characters, during transitions.
I'm just surprised no one else is complaining about it since I've lost tens of thousands of gold twice in less than 12 hours because of it and it happens at least twice a day.
Re: Scheduled Crash
I always -save right after banking transactions just to be safe, and barter with players instead of dropping items on ground/containers, but there no there is never a guarantee your character is 100% safe from disconnects/server crashes.
And yeah, it would be neat if database backups could somehow measure time spent and only run for so long before pausing, to let gameplay stay alive within a timeout limit, then continue on from where it stopped, and so forth until finished. Probably difficult to code so it can also account for new tables automatically, unless it somehow can get a runtime generated list of all tables to back up and store that list somewhere in its backup logic.
And yeah, it would be neat if database backups could somehow measure time spent and only run for so long before pausing, to let gameplay stay alive within a timeout limit, then continue on from where it stopped, and so forth until finished. Probably difficult to code so it can also account for new tables automatically, unless it somehow can get a runtime generated list of all tables to back up and store that list somewhere in its backup logic.
-
- Arelith Silver Supporter
- Posts: 735
- Joined: Mon Sep 08, 2014 2:32 am
Re: Scheduled Crash
Just resurrecting this thread to say it's still happening, just did.
Alia: The uncanny knack of Angela Amana to make the otherwise perpetually well-mannered girl lose her temper and be HORRIBLY ASHAMED afterwards.
Who on the Team to PM and When
Public Tower Discord
Tower Library
Scroll, Wand, & Potion Costs
Who on the Team to PM and When
Public Tower Discord
Tower Library
Scroll, Wand, & Potion Costs
-
- Arelith Silver Supporter
- Posts: 441
- Joined: Mon Sep 08, 2014 4:59 am
- Location: Oooooooklahoma
- Contact:
Re: Scheduled Crash
A little threadomancy but it's still happening.
Can we get an update? Is this just a "perk" that we need to plan for? Sounded like a fix would be quite involved and could cause issues down the line.
Can we get an update? Is this just a "perk" that we need to plan for? Sounded like a fix would be quite involved and could cause issues down the line.
Discord: SteelsSweets#4815
-
- Arelith Silver Supporter
- Posts: 1584
- Joined: Sun Jul 12, 2015 8:43 pm
Re: Scheduled Crash
Would it be possible to have scheduled resets right before the backups? I imagine that would help on it.
Re: Scheduled Crash
It would not make a difference in the server hanging up.
The backup functions would still go through all the necessary backup procedures regardless.
The backup functions would still go through all the necessary backup procedures regardless.
Discord: @malkalz
Determine your Public CD Key here
Can't see your vault? Have you migrated your accounts? If you have tried, and still can't see them, message me.