my experiences tuning opennms

[Adam Cassar]

Hi All,

I just wanted to post a quick note. We have a medium sized opennms
installation with about 400 nodes.

Just like everybody else we experienced problems with a high load
average - all related to disk IO. After doing all the OS tuning I could
we started to looking at the hardware. After the second upgrade to the
disk subsystem (6 disk raid 10) I had enough and did the following:

Installed a SSD card from: http://www.iomax.com/

This dropped the load from 3 to about 1.5

However the second thing we did is switch from EXT3 to XFS. This dropped
the load from 1.5 to a load average of 0.10

In summary - the SSD card helped a lot but XFS helped a lot more. It is
worth including it in your list of things to do with setting up opennms.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Christopher Chan]

Adam Cassar wrote:

> Hi All,
>
> I just wanted to post a quick note. We have a medium sized opennms
> installation with about 400 nodes.
>
> Just like everybody else we experienced problems with a high load
> average - all related to disk IO. After doing all the OS tuning I could
> we started to looking at the hardware. After the second upgrade to the
> disk subsystem (6 disk raid 10) I had enough and did the following:
>
> Installed a SSD card from: http://www.iomax.com/
>
> This dropped the load from 3 to about 1.5
>
> However the second thing we did is switch from EXT3 to XFS. This dropped
> the load from 1.5 to a load average of 0.10
>
> In summary - the SSD card helped a lot but XFS helped a lot more. It is
> worth including it in your list of things to do with setting up opennms.
>  

I don't know what hardware you are running and what version of the Linux
kernel you are running but XFS has a history of being a fantastic speed
writer that has very aggressive caching and prone to data loss after a
power cut since it does zero data journaling.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[David Hustace]

Some javascript/style in this post has been disabled (why?)

On Nov 4, 2009, at 12:21 AM, Adam Cassar wrote:

In summary - the SSD card helped a lot but XFS helped a lot more. It is 
worth including it in your list of things to do with setting up opennms.

Thanks, Adam.  What is the distro?

David

David Hustace

The OpenNMS Group, Inc.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[John A. Sullivan III]

On Wed, 2009-11-04 at 16:21 +1100, Adam Cassar wrote:

> Hi All,
>
> I just wanted to post a quick note. We have a medium sized opennms
> installation with about 400 nodes.
>
> Just like everybody else we experienced problems with a high load
> average - all related to disk IO. After doing all the OS tuning I could
> we started to looking at the hardware. After the second upgrade to the
> disk subsystem (6 disk raid 10) I had enough and did the following:
>
> Installed a SSD card from: http://www.iomax.com/
>
> This dropped the load from 3 to about 1.5
>
> However the second thing we did is switch from EXT3 to XFS. This dropped
> the load from 1.5 to a load average of 0.10
>
> In summary - the SSD card helped a lot but XFS helped a lot more. It is
> worth including it in your list of things to do with setting up opennms.

<snip>
Wow! Thanks for the tip.  Because of some of the dangers others have
mentioned with XFS, we've been shying away from it and using ext4.  Our
systems are not that heavily loaded yet to know what a difference it
will make. Does anyone have any similar comparisons between ext3 and
ext4? Thanks - John
--
John A. Sullivan III
Open Source Development Corporation
[phone]
[hidden email]

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Les Mikesell]

John A. Sullivan III wrote:

> On Wed, 2009-11-04 at 16:21 +1100, Adam Cassar wrote:
>> Hi All,
>>
>> I just wanted to post a quick note. We have a medium sized opennms
>> installation with about 400 nodes.
>>
>> Just like everybody else we experienced problems with a high load
>> average - all related to disk IO. After doing all the OS tuning I could
>> we started to looking at the hardware. After the second upgrade to the
>> disk subsystem (6 disk raid 10) I had enough and did the following:
>>
>> Installed a SSD card from: http://www.iomax.com/
>>
>> This dropped the load from 3 to about 1.5
>>
>> However the second thing we did is switch from EXT3 to XFS. This dropped
>> the load from 1.5 to a load average of 0.10
>>
>> In summary - the SSD card helped a lot but XFS helped a lot more. It is
>> worth including it in your list of things to do with setting up opennms.
> <snip>
> Wow! Thanks for the tip.  Because of some of the dangers others have
> mentioned with XFS, we've been shying away from it and using ext4.  Our
> systems are not that heavily loaded yet to know what a difference it
> will make. Does anyone have any similar comparisons between ext3 and
> ext4? Thanks - John

It is probably good timing to bring this up since RHEL/Centos 5.4 just shipped
XFS in the distribution update.  It may only be in the 64-bit version and you
can't install on it, but you should be able to add and mount XFS filesystems
now.   I am surprised that it would make that much difference, though.  Was that
the only change and was it just on the SSD card?

--
   Les Mikesell
    [hidden email]


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Christopher Chan]

> It is probably good timing to bring this up since RHEL/Centos 5.4 just shipped
> XFS in the distribution update.  It may only be in the 64-bit version and you
> can't install on it, but you should be able to add and mount XFS filesystems
> now.   I am surprised that it would make that much difference, though.  Was that
> the only change and was it just on the SSD card?
>
>  


Wow, XFS is now officially supported by Redhat? There must some serious
improvements in XFS huh?

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Les Mikesell]

Christopher Chan wrote:

> Adam Cassar wrote:
>> Hi All,
>>
>> I just wanted to post a quick note. We have a medium sized opennms
>> installation with about 400 nodes.
>>
>> Just like everybody else we experienced problems with a high load
>> average - all related to disk IO. After doing all the OS tuning I could
>> we started to looking at the hardware. After the second upgrade to the
>> disk subsystem (6 disk raid 10) I had enough and did the following:
>>
>> Installed a SSD card from: http://www.iomax.com/
>>
>> This dropped the load from 3 to about 1.5
>>
>> However the second thing we did is switch from EXT3 to XFS. This dropped
>> the load from 1.5 to a load average of 0.10
>>
>> In summary - the SSD card helped a lot but XFS helped a lot more. It is
>> worth including it in your list of things to do with setting up opennms.
>>  
>
> I don't know what hardware you are running and what version of the Linux
> kernel you are running but XFS has a history of being a fantastic speed
> writer that has very aggressive caching and prone to data loss after a
> power cut since it does zero data journaling.

Who does data journaling by default?

--
   Les Mikesell
    [hidden email]

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Christopher Chan]

Les Mikesell wrote:

> John A. Sullivan III wrote:
>  
>> On Wed, 2009-11-04 at 16:21 +1100, Adam Cassar wrote:
>>    
>>> Hi All,
>>>
>>> I just wanted to post a quick note. We have a medium sized opennms
>>> installation with about 400 nodes.
>>>
>>> Just like everybody else we experienced problems with a high load
>>> average - all related to disk IO. After doing all the OS tuning I could
>>> we started to looking at the hardware. After the second upgrade to the
>>> disk subsystem (6 disk raid 10) I had enough and did the following:
>>>
>>> Installed a SSD card from: http://www.iomax.com/
>>>
>>> This dropped the load from 3 to about 1.5
>>>
>>> However the second thing we did is switch from EXT3 to XFS. This dropped
>>> the load from 1.5 to a load average of 0.10
>>>
>>> In summary - the SSD card helped a lot but XFS helped a lot more. It is
>>> worth including it in your list of things to do with setting up opennms.
>>>      
>> <snip>
>> Wow! Thanks for the tip.  Because of some of the dangers others have
>> mentioned with XFS, we've been shying away from it and using ext4.  Our
>> systems are not that heavily loaded yet to know what a difference it
>> will make. Does anyone have any similar comparisons between ext3 and
>> ext4? Thanks - John
>>    
>
> It is probably good timing to bring this up since RHEL/Centos 5.4 just shipped
> XFS in the distribution update.  It may only be in the 64-bit version and you
> can't install on it, but you should be able to add and mount XFS filesystems
> now.   I am surprised that it would make that much difference, though.  Was that
> the only change and was it just on the SSD card?
>
>  

Yeah, it is only in the 64-bit version. But wait, there is more.

https://bugzilla.redhat.com/show_bug.cgi?id=521173


You cannot create a XFS filesystem. Or in other words, XFS is not
supported unless you give us money.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Christopher Chan]

Adam Cassar wrote:

> Hi All,
>
> I just wanted to post a quick note. We have a medium sized opennms
> installation with about 400 nodes.
>
> Just like everybody else we experienced problems with a high load
> average - all related to disk IO. After doing all the OS tuning I could
> we started to looking at the hardware. After the second upgrade to the
> disk subsystem (6 disk raid 10) I had enough and did the following:
>
> Installed a SSD card from: http://www.iomax.com/
>
> This dropped the load from 3 to about 1.5
>
> However the second thing we did is switch from EXT3 to XFS. This dropped
> the load from 1.5 to a load average of 0.10
>
> In summary - the SSD card helped a lot but XFS helped a lot more. It is
> worth including it in your list of things to do with setting up opennms.
>  

Hmm, I have over 500 nodes, I use an elcheapo whitebox, just two disks
in a software mirror, JFS and I do not have disk I/O issues (well,
certainly not noticeable even before I killed the locate database
rebuild cron job) and this box also serves as a squid proxy for around
800 users and hosts ocsinventory. Load average is 0.6 and I have not got
round to telling it not monitor nodes that are computers.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Christopher Chan]

>> I don't know what hardware you are running and what version of the Linux
>> kernel you are running but XFS has a history of being a fantastic speed
>> writer that has very aggressive caching and prone to data loss after a
>> power cut since it does zero data journaling.
>>    
>
> Who does data journaling by default?
>
>  

I did when I ran the mailservers of Outblaze Ltd (said messaging
business now part of IBM - www.lotuslive.com). Using XFS was my biggest
mistake ever. Okay, to be fair, I only had two incidents over four
years, the larger one resulting in some 4000 zero size files in the queue.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Les Mikesell]

Chan Chung Hang Christopher wrote:

>
>> It is probably good timing to bring this up since RHEL/Centos 5.4 just shipped
>> XFS in the distribution update.  It may only be in the 64-bit version and you
>> can't install on it, but you should be able to add and mount XFS filesystems
>> now.   I am surprised that it would make that much difference, though.  Was that
>> the only change and was it just on the SSD card?
>>
>>  
> Yeah, it is only in the 64-bit version. But wait, there is more.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=521173
>
>
> You cannot create a XFS filesystem. Or in other words, XFS is not
> supported unless you give us money.

If you don't want to give redhat money, you would use Centos.  They've include
the XFS utils and kernel module in their extras repo for a long time.

--
    Les Mikesell
     [hidden email]

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Alexander Finger]

Hi All,

I have done a significant amount of "IO Tuning" before we set up our systems.
Some of them can be read here: http://www.opennms.org/wiki/Stresstest_results_20060905 (is that really 3y ago?)

The result was:

XFS and Fibre Channel is the fastest.

>in a software mirror, JFS and I do not have disk I/O issues (well,

JFS was second.

EXT3 was last, even with journaling and atime switched off. The other problem with EXT3 is the limited number of files you can store in one directory. We wanted to monitor more than 25k Nodes which eliminates EXTn straight away.

Inside the netapp we found the number of disks we needed was min 19. Until 25 spindles we could measure a performance improvement, disk no 26 did not add any speed anymore.

So the "high-speed" OpenNMS System uses Fibre Channel to a SAN with +19 spindles, has separated the database from the opennms system and has split webui from OpenNMS "Core" Services.

best
Alex

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Alexander Finger]

I have updated the OpenNMS Wiki as well and invite you to correct / amend :)
http://www.opennms.org/wiki/Performance_tuning#Disk_Tuning


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Christopher Chan]

Les Mikesell wrote:

> Chan Chung Hang Christopher wrote:
>  
>>> It is probably good timing to bring this up since RHEL/Centos 5.4 just shipped
>>> XFS in the distribution update.  It may only be in the 64-bit version and you
>>> can't install on it, but you should be able to add and mount XFS filesystems
>>> now.   I am surprised that it would make that much difference, though.  Was that
>>> the only change and was it just on the SSD card?
>>>
>>>  
>>>      
>> Yeah, it is only in the 64-bit version. But wait, there is more.
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=521173
>>
>>
>> You cannot create a XFS filesystem. Or in other words, XFS is not
>> supported unless you give us money.
>>    
>
> If you don't want to give redhat money, you would use Centos.  They've include
> the XFS utils and kernel module in their extras repo for a long time.
>
>  

Oh sorry Les for leaving that out. But there is the difference that you
won't get some kernel developer to look into any problems you have but I
suppose the Centos team could try to get the attention of somebody...

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Christopher Chan]

Alexander Finger wrote:

> Hi All,
>
> I have done a significant amount of "IO Tuning" before we set up our
> systems.
> Some of them can be read here:
> http://www.opennms.org/wiki/Stresstest_results_20060905 (is that
> really 3y ago?)
>
> The result was:
>
> XFS and Fibre Channel is the fastest.
>
> >in a software mirror, JFS and I do not have disk I/O issues (well,
>
> JFS was second.

Being from an email background I look at fsbench by Bruce Guenter. A
nice chap volunteered some time and published the results of his
testing. http://www.htiweb.inf.br/benchmark/fsbench.htm

JFS was second in write performance too.


>
> EXT3 was last, even with journaling and atime switched off. The other
> problem with EXT3 is the limited number of files you can store in one
> directory. We wanted to monitor more than 25k Nodes which eliminates
> EXTn straight away.

Not surprised at all.

>
> Inside the netapp we found the number of disks we needed was min 19.
> Until 25 spindles we could measure a performance improvement, disk no
> 26 did not add any speed anymore.
>
> So the "high-speed" OpenNMS System uses Fibre Channel to a SAN with
> +19 spindles, has separated the database from the opennms system and
> has split webui from OpenNMS "Core" Services.

OOoohh, wow, how many nodes can you track with that baby?

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Adam Cassar]

Debian Lenny, 64bit.

I posted the wrong url for the SSD card I have been using as well.

http://www.fusionio.com

in Oz

http://www.iomax.com.au

David Hustace wrote:

>
> On Nov 4, 2009, at 12:21 AM, Adam Cassar wrote:
>
>> In summary - the SSD card helped a lot but XFS helped a lot more. It is
>> worth including it in your list of things to do with setting up opennms.
>
> Thanks, Adam.  What is the distro?
>
> David
>
>
> David Hustace
> The OpenNMS Group, Inc.
>

--

Adam Cassar
ICT Manager
NetRegistry
-------------------------------------------------------
o:  02 9934 0501
f:  02 9699 6088
d:  02 9934 0509
-------------------------------------------------------

Netregistry Pty Ltd
www.netregistry.com.au <http://www.netregistry.com.au>
PO Box 270 Broadway, NSW 2007
tel: [phone]  fax: [phone]



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss