UC Playerworld lag

[Cubical]

I'm experiencing some really bad serverside lag on my UC playerworld 'Serenity.' I've noticed that it is taking classes anywhere from 20-90+ seconds to load on login. I don't think it's any of my code as nothing has changed since last night when it was working fine. I've restarted the NPC-Server and that did not help. Is anyone else experiencing the same thing or have any suggestions on things to try? I have a limited amount of time each week to play around on it so when things like this happen it really hinders my ability to do anything. I just need to figure out if it's user error or an issue that others are experiencing.

edit: added /stats

PHP Code:

GServer statistics:
Server uptime: 679 hrs 14 mins
Max supported players: 16384
Sleeptime (secs/min): 59401
Connection tries (last min): 3
Players-in (byte/l.m.): 4723
Players-out (byte/l.m.): 6433
RCs-in (byte/last min): 123
RCs-out (byte/last min): 2505
NPCServer-in (byte/l.m.): 14569
NPCServer-out (byte/l.m.): 5448
ServerList-in (byte/l.m.): 4045
ServerList-out (byte/l.m.): 0
Player-avrg-in (bit/sec): 257
Player-avrg-out (bit/sec): 1472
TSockets-in (byte/l.m.): 0
TSockets-out (byte/l.m.): 0
Levels: 3
Players: 1
RCs: 2
NPCServers: 1
ServerListers: 1
Memory (bytes): 2668536
Players connecting: 0
Script engine: 0 event objects, 3 destroyed objects, 1 static objects, 6 global variables, 3 script machines
NPC-Server statistics:
Server uptime: 0 hrs 58 mins
Sleeptime (secs/min): 56515
Scriptime (last min): 56600
Connectiontries (last min): 0
NCs-in (byte/last min): 87
NCs-out (byte/last min): 0
GServer-in (byte/l.m.): 12744
GServer-out (byte/l.m.): 15007
TSockets-in (byte/l.m.): 0
TSockets-out (byte/l.m.): 0
Levels: 75
Active Levels: 13
NPCs: 195
Active Objects: 0
DB-NPCs: 5
NPC-Placeholder: 15
Projectiles: 0
Players: 3
NC count: 1
GServers: 1
Memory (bytes): 5313240
Top 10 of npcs taken most CPU time:
1. 13.443266948 % -*System/Inventory
2. 0.002356767 % -*System/Combat
3. 0.001707474 % -*System/Regeneration
4. 0.000168482 % Control-NPC
5. 0.000164906 % -*System/ImageUploader
6. 0.000023444 % npcs[200] (in level central_a-3.nw at pos (53, 10)) (object_tree)
7. 0.000016689 % npcs[180] (in level central_a-2.nw at pos (50, 0)) (object_sign)
8. 0.000008344 % npcs[181] (in level central_a-2.nw at pos (24, 10)) (object_tree)
9. 0.000006755 % npcs[202] (in level central_a-3.nw at pos (35, 23)) (object_tree)
10. 0.000006357 % npcs[208] (in level central_a-3.nw at pos (47, 61)) (object_tree)
Most used classes:
1. object_tree: 108 times
2. object_waterrock: 26 times
3. object_rock: 7 times
4. object_rock4: 7 times
5. functions_sql: 4 times
6. object_rock2: 4 times
7. object_rock3: 4 times
8. object_sign: 2 times
9. object_rock5: 2 times
10. : 1 times
Script engine: 0 event objects, 2 destroyed objects, 0 static objects, 36 global variables, 4 script machines 


/memstats

PHP Code:

Memory taken by objects: 2053240 


EDIT 2: I've determined it's an issue with the server by commenting out everything.

[fowlplay4]

It would leave me to believe some other UC server is hogging all the resources.

[Cubical]

Originally Posted by fowlplay4

It would leave me to believe some other UC server is hogging all the resources.

That is what I was assuming as well. It's currently making mine unusable. I thought there was a way on graal.us to view what servers are on the same machine but I looked and guess there isn't.

[Cubical]

There was a DBNPC that I did not comment out and I determined that the following query was causing lag out of nowhere when it's been working for months so this is more of a coding question than a bug report now.

PHP Code:



temp.query = "UPDATE player_items SET equipped='1' WHERE playeraccount='" @ player.account @ "' AND itemid='" @ itemid @ "'"; 


[Cubical]

Once again, I'm incorrect. I have now removed every piece of code from the server and I'm still experiencing the same issues. It turns out that whenever I do a query for ANYTHING in the SQLite database it locks up. Even dropping empty tables cause a 2+ minute freeze on the serverside.

[fowlplay4]

Did you try deleting your SQLite db file?

[Cubical]

Originally Posted by fowlplay4

Did you try deleting your SQLite db file?

No, it seems it only occurs when writing to the database but not requesting data from it. Also I feel like just deleting the databse isn't a very good resolution to the issue. It's just not practical if you have a live server not that it is a live classic server but I'd prefer to know whats causing it so I can take preventive measures to keep it from happening again. As a last resort I will delete the database but I just don't see it being practical.

[fowlplay4]

Well make a backup of the database but starting from a fresh one should at least help you narrow down where the issue is.

[Admins]

There were some network problems which we are working on.

[DazJenova]

Originally Posted by Stefan

There were some network problems which we are working on.

Also on RC on Era constantly getting disconnected now as well as a message spamming my RC saying "You don't have admin rights", so frustrating when scripting as I've got to exit RC.

[Cubical]

Originally Posted by Stefan

There were some network problems which we are working on.

Were they effecting just writing to a SQLite database?

[callimuc]

Originally Posted by DazJenova

Also on RC on Era constantly getting disconnected now as well as a message spamming my RC saying "You don't have admin rights", so frustrating when scripting as I've got to exit RC.

i remember that happening when i put in my email as account i think, try to enter your actual account or vice versa

[Crow]

Originally Posted by callimuc

i remember that happening when i put in my email as account i think, try to enter your actual account or vice versa

I don't think old accounts can even log in using their mail.

[shrimps]

Originally Posted by Crow

I don't think old accounts can even log in using their mail.

It lets me log in via shrimps or my email, but I don't like that as it shows my email in history window. =/

[BlueMelon]

Ok these network problems need to be fixed ASAP, I get DC'd every 10 mins, my IP gets reset to the server IP, my rights get wiped, I get the "You have no admin rights" spam... what's going on D;

[SyIvia]

*3 days later* LAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAG

oh well hopefully it will be fixed soon or is this a good excuse to make people move to igraal

[shrimps]

Never going to Iservers. o.o Kind of hard with no Idevices and just no.

[Rave_J]

idk change servers hosting lol thought how much cash flow there making they could afford there own servers to host it off of

[xXKajalXx]

Is graal down right now? there are some really bad lags and server issues going on

[GodlyAmbitions]

Originally Posted by Stefan

There were some network problems which we are working on.

Any status on this after 18 days? people still complaining hardcore, is it sure its a network problem and not a ddos? era seems to get ddosed every night and it hurts zodiac because they share the same host server.

[fowlplay4]

I sent him (Stefan) an email last night, he said he would be looking into it today or tomorrow.

These 'lagging' moments have been strictly NPC-Server, while it's lagging we're still able to PM and chat on RC (functions handled by the GServer) though so either the 'hackers' are being more specific with their 'DDoS' or there's another problem altogether.

None the less it's getting very old.

Edit: According to Stefan, Era PC was doing some very intensive **** with objects resulting in almost 3 GB of RAM usage and plenty of timeouts when players aren't around.

[BlueMelon]

Originally Posted by fowlplay4

I sent him (Stefan) an email last night, he said he would be looking into it today or tomorrow.

These 'lagging' moments have been strictly NPC-Server, while it's lagging we're still able to PM and chat on RC (functions handled by the GServer) though so either the 'hackers' are being more specific with their 'DDoS' or there's another problem altogether.

None the less it's getting very old.

Edit: According to Stefan, Era PC was doing some very intensive **** with objects resulting in almost 3 GB of RAM usage and plenty of timeouts when players aren't around.

3 GB of RAM
lolwat o_o

Maybe if we could have stefan online once in a while and tell us where the problems lie it would help out?

[xXziroXx]

Hey look at that, my NC instantly connects again when I want it too. Used to take 30+ seconds for the past year or so. I blame it entirely on Era.

[DustyPorViva]

Originally Posted by BlueMelon

Maybe if we could have stefan online once in a while and tell us where the problems lie it would help out?

I think it'd be more realistic for Stefan to set up something to notify him when a specific server is hitting such values. If one server wants to lag itself into the ground whatever, but most servers are sharing with other machines so it's not really fair for other servers.

[fowlplay4]

Originally Posted by BlueMelon

Maybe if we could have stefan online once in a while and tell us where the problems lie it would help out?

His email is [email protected]

Important things to check are pretty much anything that creates objects, classes with timeouts, anything with SQL (time your queries and see if some are slow). He said there was a bug in a house script that made classstr insanely huge whatever that means.

If you notice your server using over a GB of RAM (check /stats), savenpcs, npcshutdown, and npcstart again.

Originally Posted by DustyPorViva

I think it'd be more realistic for Stefan to set up something to notify him when a specific server is hitting such values. If one server wants to lag itself into the ground whatever, but most servers are sharing with other machines so it's not really fair for other servers.

Definitely considering this feels like it happens too often.

[Rave_J]

hey i made a post in the forums Tech selection no one even applied to it discussing the lag
http://forums.graalonline.com/forums...hp?t=134268024

[cbk1994]

Originally Posted by fowlplay4

According to Stefan, Era PC was doing some very intensive **** with objects resulting in almost 3 GB of RAM usage and plenty of timeouts when players aren't around.

This is entirely possible, I found this kind of stuff on Era many times and often suspected it was impacting other servers. Usually it's an easy fix that's caused by something stupid. For example, one time the NPC that spawns trash on the overworld created thousands of NPCs joined to "" since it was using randomstring to decide which class to spawn, and somebody had removed all except one of the trash classes from the list (recall that randomstring (used to?) have a problem with lists with only one item).

The memstats commands Stefan added are very useful in finding these things if you can get the NPC-server responsive for long enough to investigate. I think in every case it ended up with too much memory being used and was seldom a CPU problem (even though that's the first place people usually look since the CPU time listing in stats is a little misleading if you don't understand it fully).

Stefan should really look into a way to limit the amount of memory a single server can use. At the very least, it should be very vocal when exceeding a certain threshold (when I first started on Era, it was usually using 1+ GB, when I left it was closer to 250 MB, at various times I've seen it up to 3+ GB).

[Admins]

Might need to add some more limits yes.

[NicoX]

Originally Posted by Stefan

Might need to add some more limits yes.

Just shut down era

[BlueMelon]

Yes, shutdown the only PC server making him money :x

[devilsknite1]

http://statistics.graal.us/
Era should at least clean up their trash if for nothing other than courtesy of the other 7 servers it shares a host with X_x
If Era is the problem, UN has suffered greatly at Era's inefficiency. I'm not saying UN isn't causing any lag, but I am saying we don't take up 3 GB of ram from infinite loops lol

[Admins]

Most servers had some problems in the past though. It's not always easy to fix if you are not an experienced scripter, but we could may be make some guides of how to detect if somethings is taking too much cpu or memory or so.

[Rave_J]

why not just assign Skyld and Tig to go to classic world servers to check for cpu and fix the scripting issues ??

[DustyPorViva]

Originally Posted by Rave_J

why not just assign Skyld and Tig to go to classic world servers to check for cpu and fix the scripting issues ??

Monitors can be set up, why waste peoples time? Also you don't need to be a PWA to check this sort of thing out, Era's staff should be doing it.

Stefan, I'm assuming the output of /memstats is bytes? Can you output megabytes instead and specify the unit?

[fowlplay4]

it happened again.

[Crono]

lets shut era down until they stop being a nuisance

[fowlplay4]

I was about to complain again but then I logged on Era and saw Stefan online so I just am hopeful it will get fixed.

[Admins]

Ok it's not just era. The file system is a little bit under stress. I've deleted around 15 GByte of files which were not needed on Era, some other servers could also check if they really need that many logs. Also it's better to use this.dontsave = true; for temporary npcs to use the file system less often.
A few weeks ago we have also added a new backup system. It's not using a lot of CPU time but we can optimize it a little bit, we don't really need to backup the log files. So if we modify this it can possibly also speed it up a little bit.

[fowlplay4]

I didn't notice this before but when the (npcserver) is restarted the synchronous option in SQLite is set back to FULL for all the DBs which can result in a drastic performance drop when doing many inserts.

[Admins]

Yes you should add an NPC with onInitialized() which is turning it off.