Playerworlds Server Lag

[fowlplay4]

I've been getting the occasional spike on Zodiac a lot recently, I've checked our logs and stats and am unable to pinpoint a cause. I suspect someone is up to shenanigans and slowing down the server machine.

[Stephen]

Originally Posted by fowlplay4

I've been getting the occasional spike on Zodiac a lot recently, I've checked our logs and stats and am unable to pinpoint a cause. I suspect someone is up to shenanigans and slowing down the server machine.

Does Era share machines with any other playerworlds?

[Dragon551]

Originally Posted by Stephen

Does Era share machines with any other playerworlds?

Pretty sure Era, UN, Zodiac, Zone, and possibly the Login server are all on the same server.

[Tim_Rocks]

Originally Posted by fowlplay4

I've been getting the occasional spike on Zodiac a lot recently, I've checked our logs and stats and am unable to pinpoint a cause. I suspect someone is up to shenanigans and slowing down the server machine.

What kind of shenanigans are we talkin?

[Stephen]

Originally Posted by Dragon551

Pretty sure Era, UN, Zodiac, Zone, and possibly the Login server are all on the same server.

If that's accurate then the lag may be originating from Era...

[Tim_Rocks]

I'll check and see what's causing it if it's on Era. We may add you an RC since you're probably a lot better with script efficiency than I am.

Okay, so I've said /stats on RC a couple of times, and I can probably say with this data, that Era is the cause of the lag (I think??)..

PHP Code:

Server uptime: 1299 hrs 13 mins
Max supported players: 16382
Sleeptime (secs/min): 52386
Connection tries (last min): 6
Players-in (byte/l.m.): 5157586
Players-out (byte/l.m.): 13932311
RCs-in (byte/last min): 188
RCs-out (byte/last min): 373772
NPCServer-in (byte/l.m.): 4321752
NPCServer-out (byte/l.m.): 6190981
ServerList-in (byte/l.m.): 4091
ServerList-out (byte/l.m.): 38378
Player-avrg-in (bit/sec): 8674
Player-avrg-out (bit/sec): 18241
TSockets-in (byte/l.m.): 51088
TSockets-out (byte/l.m.): 318514
Levels: 268
Players: 73
RCs: 10
NPCServers: 1
ServerListers: 1
Memory (bytes): 29885912
Players connecting: 0
Script engine: 0 event objects, 69 destroyed objects, 1 static objects, 6 global variables, 3 script machines
NPC-Server statistics:
Server uptime: 1299 hrs 13 mins
Sleeptime (secs/min): 49056
Scriptime (last min): 58700
Connectiontries (last min): 0
NCs-in (byte/last min): 16
NCs-out (byte/last min): 0
GServer-in (byte/l.m.): 6167843
GServer-out (byte/l.m.): 4322143
TSockets-in (byte/l.m.): 0
TSockets-out (byte/l.m.): 0
Levels: 3092
Active Levels: 375
NPCs: 15058
Active Objects: 2169
DB-NPCs: 9401
NPC-Placeholder: 4002
Projectiles: 19
Players: 83
NC count: 5
GServers: 1
Memory (bytes): 397242528
Top 10 of npcs taken most CPU time:
1. 1.429794232 % -KillStreaks (func_numbers)
2. 0.951229333 % GangSys/Bonus (func_numbers)
3. 0.710798501 % TopKills (in level control.nw at pos (30.5, 30)) (func_time)
4. 0.693522294 % PlayerControl (in level control.nw at pos (30.5, 30))
5. 0.406092802 % -Minimap (func_tween,func_easing)
6. 0.348140001 % DB_AmmoMart (in level era_ammomartrestock.nw at pos (28, 26.5))
7. 0.328036149 % -BulletSystem (hpfunctions)
8. 0.321574211 % Control-NPC (tradefunctions,hpfunctions,partyfunctions,mail_functions)
9. 0.30520002 % DB_PyroTech (in level era_pyrotechxinside06.nw at pos (27, 17))
10. 0.267698764 % LagBot (in level control.nw at pos (30.5, 30))
Most used classes:
1. bounds: 2848 times
2. housefunctions: 1627 times
3. house_object: 999 times
4. christmaspresent: 989 times
5. xmaslights: 974 times
6. block: 954 times
7. tree: 844 times
8. func_sql: 708 times
9. func_time: 687 times
10. trash: 673 times
Script engine: 3 event objects, 2510 destroyed objects, 255 static objects, 10724 global variables, 7 script machines 


Notice the 5-6% jump
EDIT: I've checked stats a couple more times, these results don't seem to appear again..

PHP Code:

GServer statistics:
Server uptime: 1299 hrs 14 mins
Max supported players: 16382
Sleeptime (secs/min): 34055
Connection tries (last min): 6
Players-in (byte/l.m.): 4424975
Players-out (byte/l.m.): 8194827
RCs-in (byte/last min): 95
RCs-out (byte/last min): 192430
NPCServer-in (byte/l.m.): 2910074
NPCServer-out (byte/l.m.): 5256280
ServerList-in (byte/l.m.): 3979
ServerList-out (byte/l.m.): 18004
Player-avrg-in (bit/sec): 8674
Player-avrg-out (bit/sec): 18242
TSockets-in (byte/l.m.): 55681
TSockets-out (byte/l.m.): 146919
Levels: 251
Players: 71
RCs: 10
NPCServers: 1
ServerListers: 1
Memory (bytes): 29665792
Players connecting: 0
Script engine: 0 event objects, 72 destroyed objects, 1 static objects, 6 global variables, 3 script machines
NPC-Server statistics:
Server uptime: 1299 hrs 14 mins
Sleeptime (secs/min): 37968
Scriptime (last min): 45300
Connectiontries (last min): 1
NCs-in (byte/last min): 57
NCs-out (byte/last min): 12359
GServer-in (byte/l.m.): 4807242
GServer-out (byte/l.m.): 2970851
TSockets-in (byte/l.m.): 0
TSockets-out (byte/l.m.): 0
Levels: 3094
Active Levels: 373
NPCs: 14899
Active Objects: 2156
DB-NPCs: 9392
NPC-Placeholder: 3866
Projectiles: 13
Players: 81
NC count: 5
GServers: 1
Memory (bytes): 399990016
Top 10 of npcs taken most CPU time:
1. 6.772886514 % Account-System
2. 5.362913211 % -Polling (func_sql,func_time)
3. 5.266596873 % npcs[3811] (in level era_present_aa-00.nw at pos (52, 1)) (cloneplayer,func_offlineplayers)
4. 3.863579432 % npcs[2154] (in level era_present_01-00.nw at pos (44, 46)) (cloneplayer,func_offlineplayers)
5. 2.589734792 % npcs[4178] (in level era_present_aa-aa.nw at pos (47, 61)) (cloneplayer,func_offlineplayers)
6. 1.426584323 % npcs[3810] (in level era_present_aa-00.nw at pos (42, 0)) (cloneplayer,func_offlineplayers)
7. 1.153976917 % -KillStreaks (func_numbers)
8. 0.921966234 % DB_GunRack
9. 0.802496274 % GangSystem (func_sql)
10. 0.788161754 % Sales_Event (in level era_sales-test.nw at pos (32, 32)) (func_sql,func_time)
Most used classes:
1. bounds: 2848 times
2. housefunctions: 1628 times
3. house_object: 999 times
4. christmaspresent: 989 times
5. xmaslights: 974 times
6. block: 953 times
7. tree: 844 times
8. func_sql: 707 times
9. func_time: 687 times
10. trash: 686 times
Script engine: 3 event objects, 2543 destroyed objects, 327 static objects, 10787 global variables, 7 script machines 


[smirt362]

Are y'all getting a lot of lag on Era? I know the login server has been acting funny for me lately. Not loading anything when I login every once in awhile.

[Bell]

With just a quick look at Graal Stats it looks like Zodiac, UN, Era, Zone and Login are all on the same machine. Possibly more but those appear to be the main ones.

[Admins]

I've fixed a wrong use of "new TServerPlayer" on Unholy Nation, this object needs to be destroyed by script. Just call .destroy() on it (real players will not be destroyed).

[fowlplay4]

I've been receiving complaints about this happening again.

[Stephen]

Originally Posted by fowlplay4

I've been receiving complaints about this happening again.

There are huge logs on Era. For some reason -- I'm told because of their file structure -- whenever someone opens the file browser it lags the whole server for several seconds, sometimes a minute.

This may be the root of the whole problem.

[cbk1994]

Originally Posted by Stephen

There are huge logs on Era. For some reason -- I'm told because of their file structure -- whenever someone opens the file browser it lags the whole server for several seconds, sometimes a minute.

This is usually caused by some staff member giving themselves more folder rights than they need (like */*), or some new scripter who didn't know better making a bunch of new log folders rather than reusing the existing logging infrastructure. The server freezes up when file browser is opened if there are a large number of folders.

I re-did Era's file logging infrastructure a while back to avoid creating too many folders. Make sure nobody has access to logs outside of what they really need (don't give out logs/*/*/* etc. because there are old logs still there that Stefan never cleared despite me asking multiple times, make sure nobody has the accounts folder or levels/accounts/*..., etc).

As Stefan said a while back, Era really does need to cut down on logs (there are typically many gigabytes of logs stored). Unfortunately, he was never willing to actually help us by implementing log rotation or similar.

[BlueMelon]

Yes, it seems to be anyone who has access to logs on Era. I have removed the logs rights from my rights maybe a month ago, my filebrowser loads almost instantly. I have done the same with Zach's rights and a few other people too.

But for the PR's... They kinda need to open filebrowser for log checking.

Does it still lagg with client-rc I wonder?

[Crono]

lets remove logs and go back to simpler times on graal (:

[dbug]

time to clear the logs maybe?