Hi,

I am experiencing peak CPU situations of up to 500% on a virtual server here (Ubuntu, guaranteed resources 2 cores 3GHz and 4GB RAM). The last setup was 9 regions with pretty much nothing on it. Oh yes and it's configured on MySQL. I am still learning how to deal with all this - so not much content yet. The rest of the server runs some web pages with low traffic. The OpenSim.log shows stuff like this during those high load situations:

2012-06-17 11:04:39,396 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat (SnowTon01)". ThreadState=WaitSleepJoin. Last tick was 280867ms ago
2012-06-17 11:04:39,396 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat (SnowTon02)". ThreadState=WaitSleepJoin. Last tick was 210520ms ago
... and more such for all 9 regions.

My provider sais, it's the mono process which is eating up the server. Normally, with 3 avas online, teleporting and building a bit, nothing spectacular happens on the server. Maybe 10% CPU load as I would expect it. And then all of a sudden these stall situations. Anyone any idea how and where to look into deeper to find a reason?

Thank you

Phaan

Phaandor,

I've noticed that after a period of building, or debugging scripts, my sim tends to exhibit the same behavior. If not building or debugging, it'll run for long periods just fine, but after short periods of building or debugging, the sim gets overloaded.

For building and debugging, I'd recommend doing so in a Sandbox region, and then taking completed builds and debugged scripts to the final 'production' sim.

Other than that, I'm interested in knowing the cause of such overloads, as I occasionally experience similar issues when nothing else is going on.

Shad MOrde

If your hoster says it is Mono, I would believe them, Mono has pretty lousy garbage collection and does not return RAM back for system use very well.
The Watchdog messages are normal for all threads in OpenSim, but a bad script can really lock up a region.

500% is a high load for sure though, does it return to normal at all after a while?

Thanks for the replies!
> does it return to normal at all after a while?
oh yes, this behavior lasts for two to three minutes and then turns back to normal like nothing has happened. My first impression was indeed that it looks like a garbage collection on steroids.
Yesterday I reduced the setup to only one region with the default island and nothing happened for more than three hours. Then I added, one after another three more regions mostly consisting of terrain and less than ten trees in total. It took from 1pm to 9pm after which the next three minutes of 500% load occured. After that it reoccured roughly every three hours with the server idling. There seems to be a direct connection to how "tiny" the task allocation is that I give to mono. What is configured now runs with less than 200MB RAM within Opensim.exe, so this is not serious work. And again - I don't think that the server is too weak.
I wonder if there is a way to tell mono not to eat half of the datacenter of my provider while tidying up ...

Thanks

Phaan

Greetings!

I let the server running without Mono Opensim.exe from yesterday 10pm until today 3pm. As expected no CPU peaks at all. These are the peaks from the rest of the day:

Figures is date, time, CPU %, total mem, used mem, free mem (from "free"):

06/20/12 16:57:01 1,77 Mem: 4194304 941284 3253020 0 0 0
06/20/12 16:58:01 1,25 Mem: 4194304 896360 3297944 0 0 0
06/20/12 17:02:20 270,24 Mem: 4194304 1629756 2564548 0 0 0
06/20/12 17:02:20 270,24 Mem: 4194304 1693300 2501004 0 0 0
06/20/12 17:02:21 270,24 Mem: 4194304 1691264 2503040 0 0 0
06/20/12 17:02:20 270,24 Mem: 4194304 1693404 2500900 0 0 0
06/20/12 17:03:01 141,21 Mem: 4194304 874664 3319640 0 0 0
06/20/12 17:04:01 52,58 Mem: 4194304 934868 3259436 0 0 0
06/20/12 17:05:01 19,89 Mem: 4194304 951540 3242764 0 0 0
06/20/12 17:06:01 7,41 Mem: 4194304 941944 3252360 0 0 0
06/20/12 17:07:01 3,03 Mem: 4194304 930340 3263964 0 0 0
06/20/12 17:08:01 1,27 Mem: 4194304 944792 3249512 0 0 0
06/20/12 17:09:01 0,55 Mem: 4194304 947708 3246596 0 0 0
06/20/12 17:10:01 0,33 Mem: 4194304 942788 3251516 0 0 0
06/20/12 17:11:02 0,36 Mem: 4194304 1050660 3143644 0 0 0
06/20/12 17:12:01 0,41 Mem: 4194304 1032368 3161936 0 0 0
06/20/12 17:16:31 381,44 Mem: 4194304 2134732 2059572 0 0 0
06/20/12 17:16:31 381,44 Mem: 4194304 2132644 2061660 0 0 0
06/20/12 17:16:31 381,44 Mem: 4194304 2130884 2063420 0 0 0
06/20/12 17:16:31 381,44 Mem: 4194304 2129188 2065116 0 0 0
06/20/12 17:17:01 231,22 Mem: 4194304 1068468 3125836 0 0 0
06/20/12 17:18:01 85,47 Mem: 4194304 936064 3258240 0 0 0
06/20/12 17:19:01 31,95 Mem: 4194304 932252 3262052 0 0 0
06/20/12 17:20:01 13,64 Mem: 4194304 938408 3255896 0 0 0
06/20/12 17:21:01 5,14 Mem: 4194304 934432 3259872 0 0 0
06/20/12 17:22:01 1,94 Mem: 4194304 936448 3257856 0 0 0

and

06/20/12 18:29:01 0,59 Mem: 4194304 944768 3249536 0 0 0
06/20/12 18:30:01 0,64 Mem: 4194304 944944 3249360 0 0 0
06/20/12 18:35:34 511,41 Mem: 4194304 2393744 1800560 0 0 0
06/20/12 18:35:34 511,41 Mem: 4194304 2392180 1802124 0 0 0
06/20/12 18:35:34 511,41 Mem: 4194304 2393408 1800896 0 0 0
06/20/12 18:35:34 511,41 Mem: 4194304 2388516 1805788 0 0 0
06/20/12 18:35:35 511,41 Mem: 4194304 2393600 1800704 0 0 0
06/20/12 18:36:01 337,09 Mem: 4194304 927124 3267180 0 0 0
06/20/12 18:37:01 124,36 Mem: 4194304 1023616 3170688 0 0 0
06/20/12 18:38:01 46,27 Mem: 4194304 943648 3250656 0 0 0
06/20/12 18:39:01 17,09 Mem: 4194304 960288 3234016 0 0 0

I read into MONO_LOG_LEVEL environment variable but the output does not add timestamps to the messages at all. And the screen prevents redirection to a logfile. So no success there yet.

Oh and I added "--optimize=sse2" to the mono command (found in this interesting German thread: http://metropolis.hypergrid.org/forum/v ... ?f=9&t=999 ). It feels "smoother", consumes less RAM but does not help with the peaks.

Phaa

I used to have similar problems. I then changed my MaxPoolThreads from 30 to 15 and it has seemed to settle right down. Not sure if this will help you, but it has made my sims much more usable.

Ares,

unfortunately it did not help at all. I even set MaxPoolThreads down to 10. What I did find out additionally by logging also the top 4 processes whenever a peak occurs is that more than mono going way up in CPU percentage it is mysqld doing that. So as this does never occur with no Opensim.exe running, mono/Opensim is performing vast database operations during these peaks. I really really wonder if (and why, if so) I am the only one having this as my server is a pretty vanilla Ubuntu VServer. I would never have noticed it, if my provider hadn't told me. I measure this behavior by running this script every minute (it assumes that you have decimal point set instead of colon):

PROZ=`uptime | cut -d\, -f4`
DATE=`date +%D`
TIME=`date +%T`
MEM=`free | head -2 | tail -1`
echo $DATE $TIME $PROZ $MEM
ABS=`echo $PROZ | cut -d\ -f3 | cut -d\. -f1`
if test 0$ABS -gt 010
then
top -b -n 1 | head -11 | tail -5
fi

Phaa

As you know, I also experience occasional 100% CPU usage, but honestly far far less frequently than you do. I can say, however, that there is no corresponding peak for mysql when it happens, so unless you have something odd going on with your database I think you can rule mysql out as the problem.

I still think it may be some in world activity that triggers it, but I'm struggling to pin anything down, and as I said, it's an infrequent occurrence on mine.

Now, this one seems to be solved. Upgrading mono from 2.4 to 2.10 did the trick - thanks to Akira Sonoda for the decisive hint.

For whomever might be interested, this is from http://wiki.phonicuk.com/Installing-Mono-in-Ubuntu.ashx:

Ubuntu 10.04 LTS only supports Mono 2.4 'out-of-the-box'. To use version 2.6.7 (as required by McMyAdmin) you must first add the Badgerports Repository. You can do this either via the desktop, or using the command line (for installing over SSH)

Adding the repository from the command line
Add the repository to apts list of repositories:
sudo bash -c "echo deb http://badgerports.org lucid main >> /etc/apt/sources.list"
Import the key file:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 0E1FAD0C
Update your repository cache:
sudo apt-get update

Installing Mono
Once that's finished, run the following from a console:
sudo apt-get install libmono-corlib2.0-cil libmono-system-runtime2.0-cil libmono-system-web2.0-cil libmono-i18n2.0-cil libgdiplus

This results in mono 2.10.8.1 to be installed, not 2.6.7 as stated above.