Immense Lag

An area to facilitate free-form feedback on systems (in-game or out) related to Arelith.

Moderators: Forum Moderators, Active DMs, Contributors

Rook
Posts: 24
Joined: Fri May 04, 2018 1:53 am

Re: Immense Lag

Post by Rook » Sat Feb 09, 2019 10:39 pm

Started lagging on Friday again. Seems to be most severe towards the weekend? Currently, yet again, unplayable.

User avatar
Sycostick
Arelith Gold Supporter
Arelith Gold Supporter
Posts: 21
Joined: Sun Feb 14, 2016 1:14 am

Re: Immense Lag

Post by Sycostick » Sun Feb 10, 2019 1:29 am

Yes the lag is real bad today. I got on around 7:30 am est and it was laggy all morning. I just logged back on and the lag is some of the worst I have seen. Even seeing tracks stack up. There has to be dozens of tracks stacked in front of the Cordor city gates.

Tranquility
Posts: 75
Joined: Wed May 11, 2016 8:59 pm

Re: Immense Lag

Post by Tranquility » Sun Feb 10, 2019 1:12 pm

Once upon a time Arelith had issues with multiple people playing from the same ip in the same house if they didn't set it up different, totally wrecked the server. It's probably not that, but could it be sometihng like that? Lag comes and goes so strangely.

User avatar
RedGiant
Posts: 1461
Joined: Fri Sep 12, 2014 1:39 am
Location: North of Babylon

Re: Immense Lag

Post by RedGiant » Sun Feb 10, 2019 4:54 pm

I offer no theory or solutions, only this observation.

Dark Monk enters the hub in Andunor. From his pain-stakingly crafted appearance, you can tell he is a Dark Monk without him uttering a word. Every aspect about the character oozes menace. As he proceeds fearlessly before the Dhaerrow Priestesses, Titanic Ogres, and Slavering Gnolls, he emotes flawlessly. *This man's poise bespeaks an almost inhuman discipline; his every move is smooth and fluid."

Dark Monk proceeds to stutters across the hub, occasionally walking backwards-forwards, bumping into people and pillars.




True Story.
The GrumpyCat wrote:I CLICK THE HOSTIBLE BUTTON NOW U ARE DED!
Irongron wrote:The slaughter, i am afraid, will not abate.

User avatar
MissEvelyn
Arelith Silver Supporter
Arelith Silver Supporter
Posts: 1584
Joined: Sun Jul 12, 2015 8:43 pm

Re: Immense Lag

Post by MissEvelyn » Mon Feb 11, 2019 1:04 am

Cities & Planes is really bad for me at the moment of writing this.


User avatar
Yolrii
Posts: 53
Joined: Tue Jan 29, 2019 12:38 pm

Re: Immense Lag

Post by Yolrii » Mon Feb 11, 2019 2:43 am

Immense from around 5pm GMT, and it only worsened through the night. Went from pings of 2000 to 10,000 by the time I logged just now.

Completely unplayable.
Elatha Nostalfay: *she furrows her brow* "So... you lie to... everyone?"
Raice Valaine: Out of necessity, but yes, that's an excellent summary.

Freyason
Posts: 446
Joined: Tue Dec 13, 2016 12:13 am
Location: Brogendenstein

Re: Immense Lag

Post by Freyason » Mon Feb 11, 2019 3:22 am

Both Surface and C&P tonight reminded me of playing UO beta over modem, circa 1997 :D

boggle99
Posts: 87
Joined: Tue Feb 02, 2016 10:15 am

Re: Immense Lag

Post by boggle99 » Mon Feb 11, 2019 3:23 am

Post server reset on C&P which was meant to fix the lag, I'm now lagging a lot more than I was pre-reset. Not sure if it is just the fix taking a while to work or if it hasn't worked. Either way just letting you guys know.

p.s. Thanks for trying to fix it and good luck tracking down the gremlin in the system.

User avatar
Hazard
Posts: 1866
Joined: Wed Oct 24, 2018 8:27 am

Re: Immense Lag

Post by Hazard » Mon Feb 11, 2019 3:54 am

It's really weird. Seems to come and go. Totally unplayable for long periods at a time. For example it took me about 15 RL minutes to walk from Cordor's bank to the Social Club just now, and then suddenly I had absolutely no lag for about 1 minute.

User avatar
RedGiant
Posts: 1461
Joined: Fri Sep 12, 2014 1:39 am
Location: North of Babylon

Re: Immense Lag

Post by RedGiant » Wed Feb 13, 2019 2:52 am

Cities and Planes unplayable for me. I guess I am going to go watch Daredevil, season 3. This level of desperation sickens me.
The GrumpyCat wrote:I CLICK THE HOSTIBLE BUTTON NOW U ARE DED!
Irongron wrote:The slaughter, i am afraid, will not abate.

User avatar
CosmicOrderV
Arelith Silver Supporter
Arelith Silver Supporter
Posts: 389
Joined: Tue Oct 23, 2018 5:35 pm

Re: Immense Lag

Post by CosmicOrderV » Wed Feb 13, 2019 3:01 am

i cri.
Aodh Lazuli wrote:
Wed Apr 10, 2019 6:22 pm
I, too, struggle to know what is written in books without first reading them.

Mikos
Posts: 1
Joined: Fri Feb 08, 2019 9:28 pm

Re: Immense Lag

Post by Mikos » Wed Feb 13, 2019 2:36 pm

So i had really good time in the world of Arelith. But the lag issues have broken my will to continue my journey in this well done enthralling world. Not sure your servers are up to par or what the issue is but any new comers wont stay around long.I had high hopes that this world was my new homeland. The time frames and player counts have nothing to do with the issue, ive been on with 10 people on at 4 am est and i'd start rubber banding for 5 mins walking across a single zone. Restarts seem at times to make it worse... I dont have anymore time to waste on server issues that seem to be put off and only restarts happen. Did the lag issues occur before the last patch?

User avatar
Twily
Arelith Supporter
Arelith Supporter
Posts: 1120
Joined: Mon Sep 08, 2014 3:30 pm

Re: Immense Lag

Post by Twily » Wed Feb 13, 2019 4:59 pm

Mikos wrote:
Wed Feb 13, 2019 2:36 pm
So i had really good time in the world of Arelith. But the lag issues have broken my will to continue my journey in this well done enthralling world. Not sure your servers are up to par or what the issue is but any new comers wont stay around long.I had high hopes that this world was my new homeland. The time frames and player counts have nothing to do with the issue, ive been on with 10 people on at 4 am est and i'd start rubber banding for 5 mins walking across a single zone. Restarts seem at times to make it worse... I dont have anymore time to waste on server issues that seem to be put off and only restarts happen. Did the lag issues occur before the last patch?
Lag as persistently bad as we've been having recently, over this past day especially, is something I haven't seen on the server a single time in the 6+ years that I've been here.

Surface doesn't appear to be suffering from the issue either, so you could try settling in one of the Surface settlements, or just spending most of your time on the road with others in dungeons and the sort.

The best advice I can give you if you can't tolerate the lag(which is completely understandable) would be to try the server again in a few weeks, and hopefully by then the issue is resolved.

I can tell you that the devs are definitely trying to get to the bottom of the issue though, and to quote an announcement made by Spyre (HeadDM/Developer) on the Discord group:
Spyre wrote:@everyone If the servers experience immense lag, please send me a private message of the time in which it happens so I can put together a timestamp history in our logs. Thank you.

User avatar
Irongron
Server Owner/Creative Lead
Server Owner/Creative Lead
Posts: 4666
Joined: Tue Sep 09, 2014 7:13 pm

Re: Immense Lag

Post by Irongron » Wed Feb 13, 2019 5:19 pm

This is something that continues to take up a lot of our time, as we continue to investigate the issue.

The server provider is now also involved, and we will likely be taking Arelith down at some point later tonight for a more thorough investigation.

We're well aware of how frustrating this is for players, just as it is for us as developers. We'll keep everyone posted as soon as we discover anything, in the meantime I just wanted to reassure the community that this is not being ignored.

Griefmaker
Posts: 884
Joined: Mon Sep 08, 2014 8:33 pm
Location: California

Re: Immense Lag

Post by Griefmaker » Wed Feb 13, 2019 5:26 pm

Thank you for the update and thanks for all your efforts (devs/admins/DMs/etc)!

I definitely understand the frustration of something like this which seems to simply pop up without explanation.

boggle99
Posts: 87
Joined: Tue Feb 02, 2016 10:15 am

Re: Immense Lag

Post by boggle99 » Fri Feb 15, 2019 1:48 pm

Everything was fine and clear for me connecting to C&P last night. (From around 11pm till 5am GMT). :D

Sartain
Posts: 180
Joined: Fri Feb 15, 2019 3:34 pm

Re: Immense Lag

Post by Sartain » Fri Feb 15, 2019 9:57 pm

New to the place so I can't say if it's worse or better than before but we had some pretty bad lag around an hour ago, 10 PM (GMT+1)

FireGiant
Posts: 1
Joined: Sun Feb 17, 2019 4:53 pm

Re: Immense Lag

Post by FireGiant » Sun Feb 17, 2019 5:09 pm

My friends and I are new to the server, we have been here a month or so. The content is amazing, we have put in tons of hours and are very pleased with the content and the community.

That being said with the servers as they are and have been for the last few weeks we can no longer play here, as the servers are literally unplayable, constantly. As an IT professional who works daily with hardware and OS along side networking people it is hard to believe, considering the length of time this issue has persisted (for weeks now), that anything is being done to correct this issue. I realize this is not a professional outfit and once again, serious kudos on your content, truly amazing. However, your servers are literally unplayable given the current insane latency situation across all server at all times, 24/7.

I cannot help but wonder what your hardware situation is, what your backbone and ISP are doing/ not doing for you, and why such a serious issue (again, your servers are literally unplayable, and have been for weeks, the fact that anyone logs on at all speaks volumes about the content of your servers and your community) is not being addressed in the immediacy.

This is not a rant, it is with great sadness that we stop playing here. The constant 24/7 insane amounts of latency across all your servers at all times is just insurmountable, even in consideration of the great job you have done with your content. I just logged into another server just to remind myself what the game should actually play like, night and day from the experience everyone is having all the time here. I can only imagine how many other players are leaving/ losing interest over the latency. Seems like an issue you would want to solve immediately, not let persist for weeks.

User avatar
Twily
Arelith Supporter
Arelith Supporter
Posts: 1120
Joined: Mon Sep 08, 2014 3:30 pm

Re: Immense Lag

Post by Twily » Sun Feb 17, 2019 6:17 pm

I can understand if the lag is too bad for you to play.
C&P is suffering from this major lag issue, and that results in more players than would normally be present on Surface resulting in lag there during peak hours.

If you want to go to another server, I'm not going to try to stop you; although I would suggest trying out Arelith again once the issue is resolved if you feel up to it.


I do feel the need to say something over saying you find it hard to believe they're actually working on the issue as best as they can though.

I've been on the server over 6 years, and their record for dealing with issues like these is beyond exceptional considering how much work and stress they have on them.
I've never seen a lag issue take this long to be resolved, and I'd even hazard a guess that one or two of the retired developers are willing to step in to try to help as they've done in the past, if they haven't already.

I can guarantee you they are even more frustrated and stressed than you are at this ongoing issue, that they are working on it, and that it will be resolved as quickly as they are able.
That's how it's always been and how it always goes.

User avatar
Gee look who
Posts: 71
Joined: Tue Dec 18, 2018 7:03 pm

Re: Immense Lag

Post by Gee look who » Sun Feb 17, 2019 10:56 pm

At the morning (my morning at GMT-3), server was kind of empty. A DM reseted it, but the lag didn't go away.
Server was pretty empty (15 people online), so, it seems a server issue to me.

Maybe it needs a reboot, some formatting, I dont know, but this is really killing our playtime.
Please fix this so we can play at the server we love and not on those sucky other servers (as I have been forced to do).

User avatar
DM Wraith
Dungeon Master
Dungeon Master
Posts: 2202
Joined: Fri Oct 12, 2018 3:28 pm

Re: Immense Lag

Post by DM Wraith » Sun Feb 17, 2019 11:18 pm

Hi everyone,

Please remember that we are aware of immense lag issues. We are doing our best to mitigate these impacts upon players and our developers and administration is working to resolve this issue as fast as possible. Please bare with us while we continue to attempt to derive the still undetermined cause of the lag; it affects our time and resources as well, and we all want to get back to enjoying the server and community we love.

Cheers,
DM Wraith

User avatar
Irongron
Server Owner/Creative Lead
Server Owner/Creative Lead
Posts: 4666
Joined: Tue Sep 09, 2014 7:13 pm

Re: Immense Lag

Post by Irongron » Mon Feb 18, 2019 12:47 am

So after a lot of worth this evening this is currently looking very good. It is too early to call of course, but thus far signs are positive.

Players reading this can help out by hopping onto C&P so we can do a proper stress test with high numbers.

Piri
Posts: 16
Joined: Mon Jul 23, 2018 4:07 am

Re: Immense Lag

Post by Piri » Mon Feb 18, 2019 12:49 am

Thank you for your hard work.

User avatar
Mithreas
Emeritus Admin
Emeritus Admin
Posts: 2555
Joined: Sat Sep 06, 2014 3:09 am

Re: Immense Lag

Post by Mithreas » Mon Feb 18, 2019 2:07 am

FireGiant wrote:
Sun Feb 17, 2019 5:09 pm
As an IT professional who works daily with hardware and OS along side networking people it is hard to believe, considering the length of time this issue has persisted (for weeks now), that anything is being done to correct this issue. I realize this is not a professional outfit and once again, serious kudos on your content, truly amazing. However, your servers are literally unplayable given the current insane latency situation across all server at all times, 24/7.

I cannot help but wonder what your hardware situation is, what your backbone and ISP are doing/ not doing for you, and why such a serious issue (again, your servers are literally unplayable, and have been for weeks, the fact that anyone logs on at all speaks volumes about the content of your servers and your community) is not being addressed in the immediacy.
So, there are some great points in here, but also some misconceptions. Now that we seem to have solved the issue, let me unpack a few things and give some insight on what it's like to maintain Arelith from a sysadmin perspective.

There are lots of things about Arelith that are very much like any other IT environment.
  • We have a web server and SQL database, and so a lot of global best practices apply, especially around security.
  • We run on Linux, using a mix of virtual and dedicated hardware instances based on the needs of the different processes - again, very normal.
  • We have a team of a couple of dozen collaborators, and need to manage the inputs from all those people, ensure they all have the right levels of system access, etc. etc. - again, not unlike a small business' IT needs.
  • We're able to make use of industry tools like Git and TeamCity to handle the parts of our operations that are common to other use cases.
However, we also have some challenges that are more unique.
  • We have a volunteer staff. This makes change management harder - you don't want any one person being a bottleneck, but that also makes it challenging to ensure that everything that goes into the module is aligned.
  • But more importantly, we're using somebody else's software. Our main service - the game server - is closed source code maintained by Beamdog. While they're good and responsive when we raise issues with them, they are not the ones maintaining the Arelith service - we are.
  • On top of that, NWN is still very much a niche service. Unlike with almost any other application you use in IT, you can't just Google to find the solution to your problem. Arelith is literally unique in the scale of service it offers on the NWNXEE platform - we hit problems that nobody else is at risk from.
It's the second point that is probably the most interesting. As an IT professional, you're probably familiar with the usual problem with troubleshooting issues in networks with multiple components - the "finger pointing" problem where the owners of each component are adamant that their component is not at fault, because there is nothing in their diagnostics that points to any problem.

My own professional experience is right in this area - I work for a company that deploys core voice network components, essentially making telephones work, and every network is a multi-party environment. In this sort of environment, what you need is one person to take responsibility for isolating where the problem is - essentially proving which component is at fault. That's a very challenging role, where the biggest challenge is finding the right questions to ask and finding ways to answer them with the available diagnostics.

And it's here where NWN being closed source (with very limited diagnostics) is the big challenge. We're very limited in the questions we can ask and the answers we can get out.

In the end, the root cause of our problem was a scalability issue with the NWN application itself. It's an issue I have seen before, once, ironically on Arelith about 8 years ago. We applied a workaround in our server setup to mitigate the issue, but after I handed Arelith over to Irongron, this sort of setup wasn't documented anywhere. When the new team changed the server hosting setup, they didn't apply this config tweak, which made us vulnerable to the scalability issue again.

I am one of probably under 5 people in the whole world who is (a) aware of this problem and (b) knows the quick workaround to make it go away. It's not possible to find it from logs or stats (and indeed some stats are actively misleading, drawing attention away from the root cause). Last time, it took us attaching a debugger to the live server during busy hour and an expert in low level analysis to find the issue, and a sysadmin to figure out how to work around it. The active server team doesn't have either of those skill sets any more now that Liareth and I are retired, and it's very challenging for the team to even figure out what sort of help to ask for without information that points more strongly in a particular direction. The team was engaged with the hosting provider, and had done detailed analysis with them of network traces and similar, but ultimately, when the problem is deep in the bowels of NWN, even isolating the problem down to NWN is a very challenging task.

I hope that helps explain the environment a bit more. Maintaining Arelith is actually a harder task than maintaining most commercial environments; it's comparably complex to a professional service, but with a number of obstacles that make it more challenging to operate.

-Mith
xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3±1 emulated in Javascript on an Apple IIGS at a screen resolution of 1024x1.For security reasons, please leave caps lock on while browsing.

User avatar
Kreydis
Posts: 422
Joined: Sun Dec 21, 2014 3:42 am

Re: Immense Lag

Post by Kreydis » Mon Feb 18, 2019 2:16 am

Last time, it took us attaching a debugger to the live server during busy hour and an expert in low level analysis to find the issue, and a sysadmin to figure out how to work around it.
My head slightly exploded. As always, impressive work.
It's a Dwarf, no it's a Dragon, no it's a Halfling! I think.

Post Reply